Oct 30 19:21:56.785544 ip-10-0-131-147 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 30 19:21:56.785558 ip-10-0-131-147 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 30 19:21:56.785568 ip-10-0-131-147 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 30 19:21:56.785911 ip-10-0-131-147 systemd[1]: Failed to start Kubernetes Kubelet. Oct 30 19:22:06.924140 ip-10-0-131-147 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 30 19:22:06.924155 ip-10-0-131-147 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 739feb6ec75c451da723c22b7179be9c -- Oct 30 19:24:16.944654 ip-10-0-131-147 systemd[1]: Starting Kubernetes Kubelet... Oct 30 19:24:17.302143 ip-10-0-131-147 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:24:17.302143 ip-10-0-131-147 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 19:24:17.302143 ip-10-0-131-147 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:24:17.302143 ip-10-0-131-147 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:24:17.306892 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.306813 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:24:17.306892 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.306890 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:24:17.306892 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.306895 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:24:17.306990 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.306898 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:24:17.306990 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.306903 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:24:17.306990 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.306906 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:24:17.306990 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.306911 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:24:17.306990 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.306915 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:24:17.306990 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.306918 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:24:17.306990 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.306922 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:24:17.306990 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.306924 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:24:17.306990 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.306928 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:24:17.306990 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.306930 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:24:17.306990 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.306933 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:24:17.306990 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.306935 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:24:17.306990 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.306938 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:24:17.306990 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.306941 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:24:17.306990 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.306944 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:24:17.306990 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.306946 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:24:17.306990 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.306949 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:24:17.306990 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.306952 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:24:17.306990 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.306954 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:24:17.307457 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.306957 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:24:17.307457 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.306960 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:24:17.307457 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.306962 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:24:17.307457 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.306964 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:24:17.307457 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.306967 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:24:17.307457 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.306973 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:24:17.307457 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.306977 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:24:17.307457 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.306980 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:24:17.307457 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.306983 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:24:17.307457 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.306986 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:24:17.307457 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.306989 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:24:17.307457 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.306991 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:24:17.307457 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.306994 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:24:17.307457 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.306997 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:24:17.307457 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307000 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:24:17.307457 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307003 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:24:17.307457 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307005 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:24:17.307457 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307008 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:24:17.307457 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307010 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:24:17.307908 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307013 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:24:17.307908 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307016 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:24:17.307908 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307019 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:24:17.307908 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307021 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:24:17.307908 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307025 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:24:17.307908 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307027 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:24:17.307908 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307030 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:24:17.307908 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307033 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:24:17.307908 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307036 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:24:17.307908 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307038 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:24:17.307908 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307041 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:24:17.307908 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307043 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:24:17.307908 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307046 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:24:17.307908 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307048 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:24:17.307908 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307051 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:24:17.307908 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307054 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:24:17.307908 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307056 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:24:17.307908 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307059 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:24:17.307908 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307062 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:24:17.307908 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307064 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:24:17.308402 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307067 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:24:17.308402 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307071 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:24:17.308402 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307075 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:24:17.308402 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307078 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:24:17.308402 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307081 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:24:17.308402 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307083 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:24:17.308402 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307086 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:24:17.308402 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307089 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:24:17.308402 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307091 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:24:17.308402 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307094 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:24:17.308402 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307096 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:24:17.308402 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307099 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:24:17.308402 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307101 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:24:17.308402 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307104 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:24:17.308402 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307106 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:24:17.308402 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307108 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:24:17.308402 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307111 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:24:17.308402 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307114 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:24:17.308402 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307117 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:24:17.308402 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307120 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:24:17.308866 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307122 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:24:17.308866 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307125 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:24:17.308866 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307127 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:24:17.308866 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307130 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:24:17.308866 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307132 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:24:17.308866 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307135 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:24:17.308866 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307139 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:24:17.308866 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307141 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:24:17.308866 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307144 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:24:17.308866 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307146 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:24:17.308866 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307149 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:24:17.308866 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307152 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:24:17.308866 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307156 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:24:17.308866 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307158 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:24:17.308866 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307161 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:24:17.308866 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307163 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:24:17.308866 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307166 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:24:17.308866 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307169 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:24:17.308866 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307171 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:24:17.308866 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307174 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:24:17.309338 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307176 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:24:17.309338 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307179 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:24:17.309338 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307183 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:24:17.309338 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307186 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:24:17.309338 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.307188 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:24:17.309338 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308036 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:24:17.309338 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308044 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:24:17.309338 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308047 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:24:17.309338 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308050 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:24:17.309338 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308053 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:24:17.309338 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308061 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:24:17.309338 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308064 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:24:17.309338 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308067 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:24:17.309338 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308070 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:24:17.309338 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308073 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:24:17.309338 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308075 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:24:17.309338 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308078 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:24:17.309338 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308080 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:24:17.309338 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308083 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:24:17.309338 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308086 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:24:17.309844 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308089 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:24:17.309844 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308092 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:24:17.309844 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308094 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:24:17.309844 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308097 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:24:17.309844 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308100 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:24:17.309844 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308103 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:24:17.309844 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308105 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:24:17.309844 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308108 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:24:17.309844 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308110 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:24:17.309844 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308113 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:24:17.309844 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308115 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:24:17.309844 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308118 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:24:17.309844 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308120 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:24:17.309844 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308123 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:24:17.309844 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308125 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:24:17.309844 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308128 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:24:17.309844 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308130 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:24:17.309844 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308133 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:24:17.309844 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308135 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:24:17.309844 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308138 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:24:17.310327 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308140 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:24:17.310327 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308143 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:24:17.310327 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308146 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:24:17.310327 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308149 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:24:17.310327 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308151 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:24:17.310327 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308154 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:24:17.310327 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308157 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:24:17.310327 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308160 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:24:17.310327 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308163 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:24:17.310327 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308165 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:24:17.310327 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308168 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:24:17.310327 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308171 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:24:17.310327 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308173 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:24:17.310327 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308176 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:24:17.310327 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308178 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:24:17.310327 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308181 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:24:17.310327 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308184 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:24:17.310327 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308187 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:24:17.310327 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308190 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:24:17.310804 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308192 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:24:17.310804 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308195 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:24:17.310804 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308198 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:24:17.310804 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308205 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:24:17.310804 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308208 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:24:17.310804 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308210 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:24:17.310804 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308213 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:24:17.310804 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308216 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:24:17.310804 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308218 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:24:17.310804 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308221 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:24:17.310804 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308224 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:24:17.310804 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308226 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:24:17.310804 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308229 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:24:17.310804 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308233 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:24:17.310804 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308237 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:24:17.310804 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308240 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:24:17.310804 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308243 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:24:17.310804 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308246 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:24:17.310804 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308249 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:24:17.311249 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308252 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:24:17.311249 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308256 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:24:17.311249 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308261 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:24:17.311249 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308264 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:24:17.311249 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308266 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:24:17.311249 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308269 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:24:17.311249 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308272 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:24:17.311249 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308274 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:24:17.311249 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308276 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:24:17.311249 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308279 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:24:17.311249 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308282 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:24:17.311249 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308284 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:24:17.311249 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308287 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:24:17.311249 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308289 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:24:17.311249 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308292 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:24:17.311249 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308294 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:24:17.311249 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308297 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:24:17.311249 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308299 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:24:17.311249 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308302 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:24:17.311249 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308304 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:24:17.311743 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308307 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:24:17.311743 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308309 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:24:17.311743 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308311 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:24:17.311743 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308314 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:24:17.311743 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308316 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:24:17.311743 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308319 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:24:17.311743 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308321 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:24:17.311743 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308324 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:24:17.311743 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308326 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:24:17.311743 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308328 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:24:17.311743 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308331 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:24:17.311743 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308334 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:24:17.311743 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308337 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:24:17.311743 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308842 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:24:17.311743 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308848 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:24:17.311743 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308850 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:24:17.311743 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308854 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:24:17.311743 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308856 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:24:17.311743 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308860 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:24:17.311743 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308862 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:24:17.312215 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308865 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:24:17.312215 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308871 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:24:17.312215 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308874 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:24:17.312215 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308876 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:24:17.312215 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308879 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:24:17.312215 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308881 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:24:17.312215 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308884 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:24:17.312215 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308887 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:24:17.312215 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308889 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:24:17.312215 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308892 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:24:17.312215 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308895 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:24:17.312215 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308899 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:24:17.312215 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308903 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:24:17.312215 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308906 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:24:17.312215 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308908 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:24:17.312215 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308911 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:24:17.312215 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308913 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:24:17.312215 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308916 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:24:17.312215 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308918 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:24:17.312680 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308921 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:24:17.312680 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308923 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:24:17.312680 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308926 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:24:17.312680 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308928 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:24:17.312680 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308932 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:24:17.312680 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308935 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:24:17.312680 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308938 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:24:17.312680 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308940 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:24:17.312680 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308943 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:24:17.312680 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308946 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:24:17.312680 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308948 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:24:17.312680 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308951 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:24:17.312680 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308953 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:24:17.312680 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308956 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:24:17.312680 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308959 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:24:17.312680 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308962 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:24:17.312680 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308964 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:24:17.312680 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308967 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:24:17.312680 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308969 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:24:17.312680 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308972 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:24:17.313185 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308974 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:24:17.313185 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308977 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:24:17.313185 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308979 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:24:17.313185 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308982 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:24:17.313185 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308984 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:24:17.313185 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308986 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:24:17.313185 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308989 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:24:17.313185 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308991 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:24:17.313185 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308994 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:24:17.313185 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308996 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:24:17.313185 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.308999 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:24:17.313185 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.309001 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:24:17.313185 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.309004 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:24:17.313185 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.309006 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:24:17.313185 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.309009 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:24:17.313185 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.309011 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:24:17.313185 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.309014 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:24:17.313185 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.309018 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:24:17.313185 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.309021 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:24:17.313185 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.309024 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:24:17.313672 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.309026 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:24:17.313672 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.309029 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:24:17.313672 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.309031 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:24:17.313672 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.309034 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:24:17.313672 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.309036 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:24:17.313672 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.309039 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:24:17.313672 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.309042 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:24:17.313672 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.309044 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:24:17.313672 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.309047 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:24:17.313672 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.309049 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:24:17.313672 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.309052 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:24:17.313672 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.309055 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:24:17.313672 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.309057 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:24:17.313672 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.309060 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:24:17.313672 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.309062 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:24:17.313672 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.309065 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:24:17.313672 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.309067 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:24:17.313672 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.309070 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:24:17.313672 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.309072 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:24:17.314126 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.309075 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:24:17.314126 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.309077 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:24:17.314126 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.309080 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:24:17.314126 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.309083 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:24:17.314126 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.309087 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:24:17.314126 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.309090 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:24:17.314126 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.309094 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:24:17.314126 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.309096 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:24:17.314126 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.309099 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:24:17.314126 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.309102 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:24:17.314126 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.309105 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:24:17.314126 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.309108 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:24:17.314126 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.309111 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:24:17.314126 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.309113 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:24:17.314126 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.309116 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:24:17.314126 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.309119 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:24:17.314126 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.309121 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:24:17.314126 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.309124 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:24:17.314126 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.309126 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:24:17.314725 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.309129 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:24:17.314725 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.309131 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:24:17.314725 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.309836 2582 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 19:24:17.314725 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.309847 2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 19:24:17.314725 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.309853 2582 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 19:24:17.314725 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.309858 2582 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 19:24:17.314725 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.309863 2582 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 19:24:17.314725 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.309866 2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 19:24:17.314725 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.309871 2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 19:24:17.314725 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.309875 2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 19:24:17.314725 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.309878 2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 19:24:17.314725 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.309881 2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 19:24:17.314725 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.309885 2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 19:24:17.314725 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.309889 2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 19:24:17.314725 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.309892 2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 19:24:17.314725 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.309895 2582 flags.go:64] FLAG: --cgroup-root="" Oct 30 19:24:17.314725 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.309898 2582 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 19:24:17.314725 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.309901 2582 flags.go:64] FLAG: --client-ca-file="" Oct 30 19:24:17.314725 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.309904 2582 flags.go:64] FLAG: --cloud-provider="external" Oct 30 19:24:17.314725 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.309907 2582 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 19:24:17.314725 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.309916 2582 flags.go:64] FLAG: --cluster-domain="" Oct 30 19:24:17.314725 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.309919 2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 19:24:17.314725 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.309924 2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 19:24:17.315258 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.309927 2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 19:24:17.315258 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.309931 2582 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 19:24:17.315258 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.309935 2582 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 19:24:17.315258 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.309939 2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 19:24:17.315258 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.309942 2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 19:24:17.315258 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.309945 2582 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 19:24:17.315258 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.309948 2582 flags.go:64] FLAG: --contention-profiling="false" Oct 30 19:24:17.315258 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.309951 2582 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 19:24:17.315258 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.309954 2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 19:24:17.315258 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.309957 2582 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 19:24:17.315258 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.309960 2582 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 19:24:17.315258 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.309965 2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 19:24:17.315258 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.309968 2582 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 19:24:17.315258 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.309971 2582 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 19:24:17.315258 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.309974 2582 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 19:24:17.315258 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.309977 2582 flags.go:64] FLAG: --enable-server="true" Oct 30 19:24:17.315258 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.309980 2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 19:24:17.315258 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.309987 2582 flags.go:64] FLAG: --event-burst="100" Oct 30 19:24:17.315258 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.309990 2582 flags.go:64] FLAG: --event-qps="50" Oct 30 19:24:17.315258 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.309993 2582 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 19:24:17.315258 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.309996 2582 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 19:24:17.315258 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.309999 2582 flags.go:64] FLAG: --eviction-hard="" Oct 30 19:24:17.315258 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310004 2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 19:24:17.315258 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310007 2582 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 19:24:17.315861 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310010 2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 19:24:17.315861 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310013 2582 flags.go:64] FLAG: --eviction-soft="" Oct 30 19:24:17.315861 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310016 2582 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 19:24:17.315861 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310019 2582 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 19:24:17.315861 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310022 2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 19:24:17.315861 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310025 2582 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 19:24:17.315861 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310028 2582 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 19:24:17.315861 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310033 2582 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 19:24:17.315861 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310036 2582 flags.go:64] FLAG: --feature-gates="" Oct 30 19:24:17.315861 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310042 2582 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 19:24:17.315861 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310047 2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 19:24:17.315861 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310051 2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 19:24:17.315861 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310054 2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 19:24:17.315861 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310057 2582 flags.go:64] FLAG: --healthz-port="10248" Oct 30 19:24:17.315861 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310061 2582 flags.go:64] FLAG: --help="false" Oct 30 19:24:17.315861 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310064 2582 flags.go:64] FLAG: --hostname-override="ip-10-0-131-147.us-west-2.compute.internal" Oct 30 19:24:17.315861 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310067 2582 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 19:24:17.315861 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310071 2582 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 19:24:17.315861 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310074 2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 19:24:17.315861 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310077 2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 19:24:17.315861 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310081 2582 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 19:24:17.315861 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310084 2582 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 19:24:17.315861 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310087 2582 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 19:24:17.316475 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310090 2582 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 19:24:17.316475 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310093 2582 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 19:24:17.316475 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310096 2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 19:24:17.316475 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310099 2582 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 19:24:17.316475 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310101 2582 flags.go:64] FLAG: --kube-reserved="" Oct 30 19:24:17.316475 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310104 2582 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 19:24:17.316475 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310108 2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 19:24:17.316475 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310111 2582 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 19:24:17.316475 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310113 2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 19:24:17.316475 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310116 2582 flags.go:64] FLAG: --lock-file="" Oct 30 19:24:17.316475 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310119 2582 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 19:24:17.316475 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310122 2582 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 19:24:17.316475 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310125 2582 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 19:24:17.316475 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310130 2582 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 19:24:17.316475 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310133 2582 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 19:24:17.316475 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310136 2582 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 19:24:17.316475 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310142 2582 flags.go:64] FLAG: --logging-format="text" Oct 30 19:24:17.316475 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310145 2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 19:24:17.316475 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310148 2582 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 19:24:17.316475 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310152 2582 flags.go:64] FLAG: --manifest-url="" Oct 30 19:24:17.316475 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310155 2582 flags.go:64] FLAG: --manifest-url-header="" Oct 30 19:24:17.316475 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310166 2582 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 19:24:17.316475 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310170 2582 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 19:24:17.316475 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310174 2582 flags.go:64] FLAG: --max-pods="110" Oct 30 19:24:17.316475 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310177 2582 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 19:24:17.317077 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310180 2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 19:24:17.317077 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310183 2582 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 19:24:17.317077 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310186 2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 19:24:17.317077 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310189 2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 19:24:17.317077 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310192 2582 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 19:24:17.317077 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310195 2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 19:24:17.317077 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310201 2582 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 19:24:17.317077 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310204 2582 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 19:24:17.317077 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310208 2582 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 19:24:17.317077 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310210 2582 flags.go:64] FLAG: --pod-cidr="" Oct 30 19:24:17.317077 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310213 2582 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 19:24:17.317077 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310216 2582 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 19:24:17.317077 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310219 2582 flags.go:64] FLAG: --pods-per-core="0" Oct 30 19:24:17.317077 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310222 2582 flags.go:64] FLAG: --port="10250" Oct 30 19:24:17.317077 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310225 2582 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 19:24:17.317077 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310228 2582 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0c3760730cbcc881a" Oct 30 19:24:17.317077 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310231 2582 flags.go:64] FLAG: --qos-reserved="" Oct 30 19:24:17.317077 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310234 2582 flags.go:64] FLAG: --read-only-port="10255" Oct 30 19:24:17.317077 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310237 2582 flags.go:64] FLAG: --register-node="true" Oct 30 19:24:17.317077 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310240 2582 flags.go:64] FLAG: --register-with-taints="" Oct 30 19:24:17.317077 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310243 2582 flags.go:64] FLAG: --registry-burst="10" Oct 30 19:24:17.317077 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310246 2582 flags.go:64] FLAG: --registry-qps="5" Oct 30 19:24:17.317077 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310249 2582 flags.go:64] FLAG: --reserved-cpus="" Oct 30 19:24:17.317077 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310254 2582 flags.go:64] FLAG: --reserved-memory="" Oct 30 19:24:17.317077 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310258 2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 19:24:17.317077 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310261 2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 19:24:17.317705 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310266 2582 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 19:24:17.317705 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310269 2582 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 19:24:17.317705 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310272 2582 flags.go:64] FLAG: --runonce="false" Oct 30 19:24:17.317705 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310281 2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 19:24:17.317705 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310284 2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 19:24:17.317705 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310287 2582 flags.go:64] FLAG: --seccomp-default="false" Oct 30 19:24:17.317705 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310290 2582 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 19:24:17.317705 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310293 2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 19:24:17.317705 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310296 2582 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 19:24:17.317705 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310300 2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 19:24:17.317705 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310303 2582 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 19:24:17.317705 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310306 2582 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 19:24:17.317705 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310309 2582 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 19:24:17.317705 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310312 2582 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 19:24:17.317705 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310315 2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 19:24:17.317705 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310318 2582 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 19:24:17.317705 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310321 2582 flags.go:64] FLAG: --system-cgroups="" Oct 30 19:24:17.317705 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310323 2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 19:24:17.317705 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310329 2582 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 19:24:17.317705 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310332 2582 flags.go:64] FLAG: --tls-cert-file="" Oct 30 19:24:17.317705 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310335 2582 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 19:24:17.317705 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310342 2582 flags.go:64] FLAG: --tls-min-version="" Oct 30 19:24:17.317705 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310345 2582 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 19:24:17.317705 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310357 2582 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 19:24:17.317705 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310361 2582 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 19:24:17.318319 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310363 2582 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 19:24:17.318319 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310366 2582 flags.go:64] FLAG: --v="2" Oct 30 19:24:17.318319 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310371 2582 flags.go:64] FLAG: --version="false" Oct 30 19:24:17.318319 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310375 2582 flags.go:64] FLAG: --vmodule="" Oct 30 19:24:17.318319 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310380 2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 19:24:17.318319 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310384 2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 19:24:17.318319 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310562 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:24:17.318319 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310568 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:24:17.318319 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310571 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:24:17.318319 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310574 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:24:17.318319 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310577 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:24:17.318319 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310585 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:24:17.318319 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310588 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:24:17.318319 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310591 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:24:17.318319 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310603 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:24:17.318319 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310606 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:24:17.318319 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310609 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:24:17.318319 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310611 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:24:17.318319 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310614 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:24:17.318319 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310617 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:24:17.318319 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310619 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:24:17.318839 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310622 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:24:17.318839 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310625 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:24:17.318839 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310628 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:24:17.318839 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310631 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:24:17.318839 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310633 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:24:17.318839 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310636 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:24:17.318839 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310639 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:24:17.318839 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310642 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:24:17.318839 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310644 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:24:17.318839 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310647 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:24:17.318839 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310650 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:24:17.318839 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310652 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:24:17.318839 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310655 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:24:17.318839 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310657 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:24:17.318839 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310660 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:24:17.318839 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310664 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:24:17.318839 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310668 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:24:17.318839 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310672 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:24:17.318839 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310677 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:24:17.319280 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310680 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:24:17.319280 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310683 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:24:17.319280 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310686 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:24:17.319280 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310689 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:24:17.319280 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310698 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:24:17.319280 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310702 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:24:17.319280 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310705 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:24:17.319280 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310707 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:24:17.319280 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310710 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:24:17.319280 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310712 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:24:17.319280 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310715 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:24:17.319280 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310717 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:24:17.319280 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310720 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:24:17.319280 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310723 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:24:17.319280 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310725 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:24:17.319280 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310728 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:24:17.319280 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310731 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:24:17.319280 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310733 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:24:17.319280 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310736 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:24:17.319280 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310738 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:24:17.319769 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310746 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:24:17.319769 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310749 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:24:17.319769 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310752 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:24:17.319769 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310754 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:24:17.319769 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310757 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:24:17.319769 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310759 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:24:17.319769 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310762 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:24:17.319769 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310764 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:24:17.319769 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310768 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:24:17.319769 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310771 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:24:17.319769 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310773 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:24:17.319769 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310778 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:24:17.319769 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310780 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:24:17.319769 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310783 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:24:17.319769 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310785 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:24:17.319769 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310788 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:24:17.319769 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310790 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:24:17.319769 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310799 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:24:17.319769 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310802 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:24:17.320216 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310805 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:24:17.320216 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310807 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:24:17.320216 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310810 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:24:17.320216 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310813 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:24:17.320216 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310815 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:24:17.320216 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310818 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:24:17.320216 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310820 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:24:17.320216 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310823 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:24:17.320216 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310826 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:24:17.320216 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310828 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:24:17.320216 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310831 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:24:17.320216 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310833 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:24:17.320216 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310837 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:24:17.320216 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310841 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:24:17.320216 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310844 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:24:17.320216 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310847 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:24:17.320216 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310850 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:24:17.320216 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310852 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:24:17.320216 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310855 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:24:17.320670 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310858 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:24:17.320670 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310860 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:24:17.320670 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310863 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:24:17.320670 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310866 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:24:17.320670 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310868 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:24:17.320670 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310873 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:24:17.320670 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310875 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:24:17.320670 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310878 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:24:17.320670 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310880 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:24:17.320670 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310883 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:24:17.320670 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310885 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:24:17.320670 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310888 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:24:17.320670 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310896 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:24:17.320670 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310898 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:24:17.320670 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.310927 2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:24:17.320670 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310976 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:24:17.321050 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310980 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:24:17.321050 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310984 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:24:17.321050 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310986 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:24:17.321050 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310989 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:24:17.321050 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310992 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:24:17.321050 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310994 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:24:17.321050 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310997 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:24:17.321050 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.310999 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:24:17.321050 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311002 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:24:17.321050 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311004 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:24:17.321050 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311007 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:24:17.321050 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311010 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:24:17.321050 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311012 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:24:17.321050 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311015 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:24:17.321050 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311017 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:24:17.321050 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311020 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:24:17.321050 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311022 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:24:17.321050 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311025 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:24:17.321050 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311027 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:24:17.321050 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311030 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:24:17.321532 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311032 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:24:17.321532 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311036 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:24:17.321532 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311038 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:24:17.321532 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311041 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:24:17.321532 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311043 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:24:17.321532 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311047 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:24:17.321532 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311050 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:24:17.321532 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311053 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:24:17.321532 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311056 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:24:17.321532 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311064 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:24:17.321532 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311067 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:24:17.321532 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311069 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:24:17.321532 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311072 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:24:17.321532 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311075 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:24:17.321532 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311077 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:24:17.321532 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311080 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:24:17.321532 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311082 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:24:17.321532 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311085 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:24:17.321532 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311087 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:24:17.321980 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311090 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:24:17.321980 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311092 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:24:17.321980 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311095 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:24:17.321980 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311097 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:24:17.321980 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311100 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:24:17.321980 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311102 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:24:17.321980 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311105 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:24:17.321980 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311107 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:24:17.321980 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311110 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:24:17.321980 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311112 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:24:17.321980 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311114 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:24:17.321980 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311117 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:24:17.321980 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311120 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:24:17.321980 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311122 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:24:17.321980 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311126 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:24:17.321980 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311128 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:24:17.321980 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311131 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:24:17.321980 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311134 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:24:17.321980 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311137 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:24:17.322435 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311139 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:24:17.322435 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311142 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:24:17.322435 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311144 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:24:17.322435 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311147 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:24:17.322435 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311156 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:24:17.322435 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311159 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:24:17.322435 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311161 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:24:17.322435 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311164 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:24:17.322435 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311167 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:24:17.322435 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311169 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:24:17.322435 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311172 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:24:17.322435 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311174 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:24:17.322435 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311176 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:24:17.322435 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311179 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:24:17.322435 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311181 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:24:17.322435 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311184 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:24:17.322435 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311186 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:24:17.322435 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311189 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:24:17.322435 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311191 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:24:17.322435 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311194 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:24:17.322899 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311196 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:24:17.322899 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311200 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:24:17.322899 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311204 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:24:17.322899 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311206 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:24:17.322899 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311209 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:24:17.322899 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311212 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:24:17.322899 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311214 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:24:17.322899 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311218 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:24:17.322899 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311220 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:24:17.322899 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311223 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:24:17.322899 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311225 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:24:17.322899 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311228 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:24:17.322899 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311230 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:24:17.322899 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311233 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:24:17.322899 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311235 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:24:17.322899 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311238 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:24:17.322899 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311240 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:24:17.322899 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311251 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:24:17.322899 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311255 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:24:17.322899 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311257 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:24:17.323407 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311260 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:24:17.323407 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311262 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:24:17.323407 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311265 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:24:17.323407 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311268 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:24:17.323407 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311270 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:24:17.323407 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311274 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:24:17.323407 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.311276 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:24:17.323407 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.317801 2582 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 19:24:17.323407 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.317816 2582 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 19:24:17.323407 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.317826 2582 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 19:24:17.323407 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.317831 2582 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 19:24:17.323407 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.317892 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:24:17.323407 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.317897 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:24:17.323407 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.317900 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:24:17.323407 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.317903 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:24:17.323407 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.317907 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:24:17.323407 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.317909 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:24:17.323407 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.317912 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:24:17.323407 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.317914 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:24:17.323859 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.317917 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:24:17.323859 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.317920 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:24:17.323859 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.317922 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:24:17.323859 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.317925 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:24:17.323859 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.317928 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:24:17.323859 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.317931 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:24:17.323859 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.317933 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:24:17.323859 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.317936 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:24:17.323859 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.317938 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:24:17.323859 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.317941 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:24:17.323859 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.317944 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:24:17.323859 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.317947 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:24:17.323859 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.317950 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:24:17.323859 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.317953 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:24:17.323859 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.317956 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:24:17.323859 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.317958 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:24:17.323859 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.317961 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:24:17.323859 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.317963 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:24:17.323859 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.317966 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:24:17.323859 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.317969 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:24:17.324315 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.317972 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:24:17.324315 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.317975 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:24:17.324315 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.317978 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:24:17.324315 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.317980 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:24:17.324315 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.317983 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:24:17.324315 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.317986 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:24:17.324315 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.317989 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:24:17.324315 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.317991 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:24:17.324315 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.317994 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:24:17.324315 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.317996 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:24:17.324315 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.317999 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:24:17.324315 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318001 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:24:17.324315 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318004 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:24:17.324315 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318006 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:24:17.324315 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318009 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:24:17.324315 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318011 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:24:17.324315 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318014 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:24:17.324315 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318016 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:24:17.324315 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318018 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:24:17.324844 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318021 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:24:17.324844 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318023 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:24:17.324844 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318026 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:24:17.324844 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318029 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:24:17.324844 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318032 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:24:17.324844 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318035 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:24:17.324844 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318037 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:24:17.324844 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318040 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:24:17.324844 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318043 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:24:17.324844 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318045 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:24:17.324844 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318048 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:24:17.324844 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318050 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:24:17.324844 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318053 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:24:17.324844 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318056 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:24:17.324844 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318059 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:24:17.324844 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318062 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:24:17.324844 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318064 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:24:17.324844 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318067 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:24:17.324844 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318069 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:24:17.324844 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318072 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:24:17.325313 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318075 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:24:17.325313 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318077 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:24:17.325313 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318081 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:24:17.325313 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318085 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:24:17.325313 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318088 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:24:17.325313 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318091 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:24:17.325313 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318094 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:24:17.325313 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318096 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:24:17.325313 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318099 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:24:17.325313 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318101 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:24:17.325313 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318104 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:24:17.325313 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318106 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:24:17.325313 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318108 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:24:17.325313 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318111 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:24:17.325313 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318113 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:24:17.325313 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318116 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:24:17.325313 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318119 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:24:17.325313 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318122 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:24:17.325313 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318125 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:24:17.325313 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318128 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:24:17.325780 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318130 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:24:17.325780 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318133 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:24:17.325780 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318137 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:24:17.325780 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318142 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:24:17.325780 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318146 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:24:17.325780 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318150 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:24:17.325780 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318153 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:24:17.325780 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318156 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:24:17.325780 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318159 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:24:17.325780 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318161 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:24:17.325780 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318164 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:24:17.325780 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318167 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:24:17.325780 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318171 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:24:17.325780 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318175 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:24:17.325780 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318179 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:24:17.325780 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318182 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:24:17.325780 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318184 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:24:17.325780 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318187 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:24:17.325780 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318189 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:24:17.326227 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.318217 2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:24:17.326227 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318371 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:24:17.326227 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318377 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:24:17.326227 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318380 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:24:17.326227 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318383 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:24:17.326227 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318385 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:24:17.326227 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318388 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:24:17.326227 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318391 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:24:17.326227 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318394 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:24:17.326227 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318397 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:24:17.326227 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318401 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:24:17.326227 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318405 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:24:17.326227 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318408 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:24:17.326227 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318411 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:24:17.326227 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318414 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:24:17.326227 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318416 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:24:17.326635 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318419 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:24:17.326635 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318422 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:24:17.326635 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318425 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:24:17.326635 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318428 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:24:17.326635 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318431 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:24:17.326635 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318433 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:24:17.326635 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318436 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:24:17.326635 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318439 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:24:17.326635 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318441 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:24:17.326635 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318444 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:24:17.326635 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318447 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:24:17.326635 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318449 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:24:17.326635 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318452 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:24:17.326635 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318454 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:24:17.326635 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318457 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:24:17.326635 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318459 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:24:17.326635 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318461 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:24:17.326635 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318464 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:24:17.326635 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318466 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:24:17.326635 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318469 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:24:17.327136 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318471 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:24:17.327136 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318474 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:24:17.327136 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318476 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:24:17.327136 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318479 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:24:17.327136 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318481 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:24:17.327136 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318483 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:24:17.327136 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318486 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:24:17.327136 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318488 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:24:17.327136 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318492 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:24:17.327136 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318495 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:24:17.327136 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318499 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:24:17.327136 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318502 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:24:17.327136 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318504 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:24:17.327136 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318507 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:24:17.327136 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318510 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:24:17.327136 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318512 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:24:17.327136 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318515 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:24:17.327136 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318518 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:24:17.327136 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318521 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:24:17.327861 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318523 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:24:17.327861 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318526 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:24:17.327861 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318528 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:24:17.327861 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318531 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:24:17.327861 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318533 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:24:17.327861 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318536 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:24:17.327861 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318538 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:24:17.327861 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318541 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:24:17.327861 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318543 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:24:17.327861 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318546 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:24:17.327861 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318548 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:24:17.327861 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318551 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:24:17.327861 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318553 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:24:17.327861 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318555 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:24:17.327861 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318558 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:24:17.327861 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318560 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:24:17.327861 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318563 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:24:17.327861 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318565 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:24:17.327861 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318568 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:24:17.327861 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318570 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:24:17.328640 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318573 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:24:17.328640 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318575 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:24:17.328640 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318578 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:24:17.328640 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318581 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:24:17.328640 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318584 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:24:17.328640 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318586 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:24:17.328640 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318589 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:24:17.328640 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318592 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:24:17.328640 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318601 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:24:17.328640 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318604 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:24:17.328640 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318607 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:24:17.328640 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318609 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:24:17.328640 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318612 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:24:17.328640 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318614 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:24:17.328640 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318617 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:24:17.328640 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318619 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:24:17.328640 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318622 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:24:17.328640 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318625 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:24:17.328640 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318627 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:24:17.329234 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318629 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:24:17.329234 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318632 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:24:17.329234 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318634 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:24:17.329234 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318637 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:24:17.329234 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318640 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:24:17.329234 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318642 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:24:17.329234 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318645 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:24:17.329234 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318648 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:24:17.329234 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318650 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:24:17.329234 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318653 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:24:17.329234 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318655 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:24:17.329234 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318658 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:24:17.329234 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318660 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:24:17.329234 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.318683 2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:24:17.329234 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318717 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:24:17.329234 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318721 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:24:17.329660 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318725 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:24:17.329660 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318727 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:24:17.329660 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318731 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:24:17.329660 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318734 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:24:17.329660 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318737 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:24:17.329660 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318739 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:24:17.329660 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318742 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:24:17.329660 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318745 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:24:17.329660 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318747 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:24:17.329660 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318750 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:24:17.329660 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318753 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:24:17.329660 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318755 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:24:17.329660 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318758 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:24:17.329660 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318760 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:24:17.329660 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318763 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:24:17.329660 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318765 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:24:17.329660 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318768 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:24:17.329660 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318770 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:24:17.329660 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318773 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:24:17.330125 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318775 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:24:17.330125 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318778 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:24:17.330125 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318780 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:24:17.330125 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318783 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:24:17.330125 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318785 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:24:17.330125 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318788 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:24:17.330125 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318790 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:24:17.330125 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318793 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:24:17.330125 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318795 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:24:17.330125 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318798 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:24:17.330125 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318801 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:24:17.330125 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318804 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:24:17.330125 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318806 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:24:17.330125 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318809 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:24:17.330125 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318812 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:24:17.330125 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318815 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:24:17.330125 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318817 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:24:17.330125 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318820 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:24:17.330125 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318823 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:24:17.330125 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318825 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:24:17.330711 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318828 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:24:17.330711 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318831 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:24:17.330711 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318835 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:24:17.330711 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318837 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:24:17.330711 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318840 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:24:17.330711 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318843 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:24:17.330711 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318846 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:24:17.330711 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318849 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:24:17.330711 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318851 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:24:17.330711 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318854 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:24:17.330711 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318857 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:24:17.330711 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318859 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:24:17.330711 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318862 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:24:17.330711 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318864 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:24:17.330711 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318867 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:24:17.330711 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318869 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:24:17.330711 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318872 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:24:17.330711 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318874 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:24:17.330711 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318877 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:24:17.331230 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318879 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:24:17.331230 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318881 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:24:17.331230 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318884 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:24:17.331230 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318886 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:24:17.331230 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318889 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:24:17.331230 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318891 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:24:17.331230 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318894 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:24:17.331230 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318896 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:24:17.331230 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318899 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:24:17.331230 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318902 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:24:17.331230 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318905 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:24:17.331230 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318907 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:24:17.331230 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318909 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:24:17.331230 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318915 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:24:17.331230 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318917 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:24:17.331230 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318920 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:24:17.331230 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318923 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:24:17.331230 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318926 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:24:17.331230 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318931 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:24:17.331230 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318934 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:24:17.331738 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318937 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:24:17.331738 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318940 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:24:17.331738 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318942 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:24:17.331738 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318945 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:24:17.331738 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318947 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:24:17.331738 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318950 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:24:17.331738 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318952 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:24:17.331738 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318955 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:24:17.331738 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318957 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:24:17.331738 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318960 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:24:17.331738 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318962 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:24:17.331738 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318965 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:24:17.331738 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318967 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:24:17.331738 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318972 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:24:17.331738 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318974 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:24:17.331738 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318977 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:24:17.331738 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318979 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:24:17.331738 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318982 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:24:17.331738 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318984 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:24:17.332185 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318987 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:24:17.332185 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318990 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:24:17.332185 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318992 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:24:17.332185 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318995 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:24:17.332185 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.318998 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:24:17.332185 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.319000 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:24:17.332185 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:17.319004 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:24:17.332185 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.319053 2582 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 19:24:17.332185 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.322508 2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 19:24:17.332185 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.323333 2582 server.go:1014] "Starting client certificate rotation" Oct 30 19:24:17.332185 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.323447 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 19:24:17.332185 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.323481 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 19:24:17.332185 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.325854 2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 19:24:17.332185 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.329066 2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 19:24:17.337916 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.337901 2582 log.go:25] "Validated CRI v1 runtime API" Oct 30 19:24:17.340948 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.340932 2582 log.go:25] "Validated CRI v1 image API" Oct 30 19:24:17.341805 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.341783 2582 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 19:24:17.344305 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.344278 2582 fs.go:135] Filesystem UUIDs: map[3b1dc3af-4f73-49d0-9636-5555f06fcc95:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 a38366c6-7060-434d-b947-931174199099:/dev/nvme0n1p4] Oct 30 19:24:17.344381 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.344306 2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 19:24:17.349078 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.348965 2582 manager.go:217] Machine: {Timestamp:2026-10-30 19:24:17.348483275 +0000 UTC m=+0.297662856 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099678 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec276b0d291c32218087472ba1636302 SystemUUID:ec276b0d-291c-3221-8087-472ba1636302 BootID:739feb6e-c75c-451d-a723-c22b7179be9c Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:f1:e1:59:05 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:f1:e1:59:05 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:be:f0:d4:b5:b0:3d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 19:24:17.349531 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.349521 2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 19:24:17.349628 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.349612 2582 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 19:24:17.352159 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.352137 2582 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 19:24:17.352285 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.352161 2582 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-147.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 19:24:17.352868 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.352857 2582 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 19:24:17.352868 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.352868 2582 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 19:24:17.352930 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.352895 2582 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 19:24:17.352930 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.352908 2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 19:24:17.353650 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.353641 2582 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 19:24:17.357464 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.357452 2582 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 19:24:17.357585 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.357576 2582 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 19:24:17.358261 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.358252 2582 kubelet.go:493] "Attempting to sync node with API server" Oct 30 19:24:17.358289 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.358265 2582 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 19:24:17.358289 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.358278 2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 19:24:17.358289 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.358290 2582 kubelet.go:405] "Adding apiserver pod source" Oct 30 19:24:17.358391 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.358299 2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 19:24:17.358613 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.358601 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 19:24:17.360412 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.360393 2582 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 19:24:17.363893 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.363880 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 19:24:17.364638 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.364626 2582 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 19:24:17.364683 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.364657 2582 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 19:24:17.365180 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.365170 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 19:24:17.365209 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.365187 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 19:24:17.365209 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.365193 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 19:24:17.365209 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.365199 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 19:24:17.365209 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.365204 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 19:24:17.365308 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.365211 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 19:24:17.365654 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.365641 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 19:24:17.365688 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.365657 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 19:24:17.365688 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.365675 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 19:24:17.365688 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.365686 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 19:24:17.365762 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.365708 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 19:24:17.365762 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.365724 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 19:24:17.366113 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.366104 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 19:24:17.366141 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.366114 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 19:24:17.369620 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.369607 2582 server.go:1263] "Started kubelet" Oct 30 19:24:17.369722 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.369696 2582 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 19:24:17.369778 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.369704 2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 19:24:17.369778 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.369773 2582 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 19:24:17.370312 ip-10-0-131-147 systemd[1]: Started Kubernetes Kubelet. Oct 30 19:24:17.370947 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.370933 2582 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 19:24:17.374092 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.373144 2582 server.go:317] "Adding debug handlers to kubelet server" Oct 30 19:24:17.377925 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:17.377905 2582 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 19:24:17.379067 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.379050 2582 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 19:24:17.379067 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.379064 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 19:24:17.379238 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:17.379199 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-147.us-west-2.compute.internal\" not found" Oct 30 19:24:17.379782 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.379658 2582 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 19:24:17.379782 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.379676 2582 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 19:24:17.379782 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.379744 2582 reconstruct.go:97] "Volume reconstruction finished" Oct 30 19:24:17.379782 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.379754 2582 reconciler.go:29] "Reconciler: start to sync state" Oct 30 19:24:17.379782 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.379779 2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 19:24:17.380025 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.379849 2582 factory.go:55] Registering systemd factory Oct 30 19:24:17.380025 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.379862 2582 factory.go:223] Registration of the systemd container factory successfully Oct 30 19:24:17.380127 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.380057 2582 factory.go:153] Registering CRI-O factory Oct 30 19:24:17.380127 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.380071 2582 factory.go:223] Registration of the crio container factory successfully Oct 30 19:24:17.380219 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.380140 2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 19:24:17.380219 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.380154 2582 factory.go:103] Registering Raw factory Oct 30 19:24:17.380219 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.380164 2582 manager.go:1199] Started watching for new ooms in manager Oct 30 19:24:17.381049 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.381032 2582 manager.go:319] Starting recovery of all containers Oct 30 19:24:17.392480 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.392460 2582 manager.go:324] Recovery completed Oct 30 19:24:17.398059 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.397679 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:24:17.399430 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.399414 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-147.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:24:17.399507 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.399443 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-147.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:24:17.399507 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.399457 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-147.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 19:24:17.399988 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.399974 2582 cpu_manager.go:226] "Starting" policy="none" Oct 30 19:24:17.399988 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.399985 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:24:17.400096 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.399986 2582 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 19:24:17.400096 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.400025 2582 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 19:24:17.400695 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.400680 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-147.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:24:17.400775 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.400708 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-147.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:24:17.400775 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.400723 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-147.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 19:24:17.402211 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.402196 2582 policy_none.go:50] "Start" Oct 30 19:24:17.402211 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.402213 2582 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 19:24:17.402317 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.402223 2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 19:24:17.404207 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.404197 2582 policy_none.go:44] "Start" Oct 30 19:24:17.448931 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.448898 2582 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 19:24:17.451312 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.451298 2582 manager.go:341] "Starting Device Plugin manager" Oct 30 19:24:17.451749 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:17.451734 2582 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 19:24:17.451749 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.451749 2582 server.go:87] "Starting device plugin registration server" Oct 30 19:24:17.451976 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.451965 2582 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 19:24:17.452020 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.451976 2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 19:24:17.452500 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.452480 2582 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 19:24:17.452588 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.452577 2582 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 19:24:17.452632 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.452590 2582 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 19:24:17.453512 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:17.453493 2582 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 19:24:17.453611 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:17.453591 2582 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-147.us-west-2.compute.internal\" not found" Oct 30 19:24:17.514151 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.514127 2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-147.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 19:24:17.514151 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:17.514125 2582 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-147.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 19:24:17.516130 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:17.514133 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-147.us-west-2.compute.internal.18e3651e48debf6e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-147.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-147.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 19:24:17.369571182 +0000 UTC m=+0.318750762,LastTimestamp:2026-10-30 19:24:17.369571182 +0000 UTC m=+0.318750762,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-147.us-west-2.compute.internal,}" Oct 30 19:24:17.516130 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.515114 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 19:24:17.516430 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.516410 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 19:24:17.516532 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.516436 2582 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 19:24:17.516532 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.516458 2582 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 19:24:17.516532 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:17.516496 2582 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 19:24:17.552912 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.552845 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:24:17.553671 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.553650 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-147.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:24:17.553740 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.553679 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-147.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:24:17.553740 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.553689 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-147.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 19:24:17.553740 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.553711 2582 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-147.us-west-2.compute.internal" Oct 30 19:24:17.589455 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.589428 2582 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 19:24:17.589563 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.589499 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8ksgr" Oct 30 19:24:17.617119 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.617092 2582 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-147.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-147.us-west-2.compute.internal"] Oct 30 19:24:17.617239 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.617153 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:24:17.618021 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.618004 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-147.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:24:17.618113 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.618040 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-147.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:24:17.618113 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.618055 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-147.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 19:24:17.620271 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.620256 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:24:17.620430 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.620415 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-147.us-west-2.compute.internal" Oct 30 19:24:17.620498 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.620451 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:24:17.620991 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.620974 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-147.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:24:17.621052 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.621008 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-147.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:24:17.621052 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.621019 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-147.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 19:24:17.621111 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.621006 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-147.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:24:17.621146 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.621104 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-147.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:24:17.621146 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.621128 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-147.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 19:24:17.623079 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.623057 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-147.us-west-2.compute.internal" Oct 30 19:24:17.623162 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.623092 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:24:17.623870 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.623857 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-147.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:24:17.623945 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.623891 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-147.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:24:17.623945 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.623904 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-147.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 19:24:17.644776 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:17.644756 2582 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-147.us-west-2.compute.internal\" not found" node="ip-10-0-131-147.us-west-2.compute.internal" Oct 30 19:24:17.646711 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:17.646696 2582 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-147.us-west-2.compute.internal\" not found" node="ip-10-0-131-147.us-west-2.compute.internal" Oct 30 19:24:17.670740 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.670724 2582 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:24:17.672123 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.672101 2582 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:24:17.682198 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.682177 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4cac457219e3215c4459c18e1cf7554f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-147.us-west-2.compute.internal\" (UID: \"4cac457219e3215c4459c18e1cf7554f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-147.us-west-2.compute.internal" Oct 30 19:24:17.682262 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.682201 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4cac457219e3215c4459c18e1cf7554f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-147.us-west-2.compute.internal\" (UID: \"4cac457219e3215c4459c18e1cf7554f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-147.us-west-2.compute.internal" Oct 30 19:24:17.682262 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.682220 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/799ed54bc75d16c836f178a66c89fbdc-config\") pod \"kube-apiserver-proxy-ip-10-0-131-147.us-west-2.compute.internal\" (UID: \"799ed54bc75d16c836f178a66c89fbdc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-147.us-west-2.compute.internal" Oct 30 19:24:17.764644 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.764614 2582 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:24:17.765625 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.765607 2582 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:24:17.765891 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.765874 2582 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-147.us-west-2.compute.internal" Oct 30 19:24:17.765939 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:17.765898 2582 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-147.us-west-2.compute.internal\": node \"ip-10-0-131-147.us-west-2.compute.internal\" not found" Oct 30 19:24:17.779738 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.779719 2582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-147.us-west-2.compute.internal" Oct 30 19:24:17.782604 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.782583 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4cac457219e3215c4459c18e1cf7554f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-147.us-west-2.compute.internal\" (UID: \"4cac457219e3215c4459c18e1cf7554f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-147.us-west-2.compute.internal" Oct 30 19:24:17.782672 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.782658 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4cac457219e3215c4459c18e1cf7554f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-147.us-west-2.compute.internal\" (UID: \"4cac457219e3215c4459c18e1cf7554f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-147.us-west-2.compute.internal" Oct 30 19:24:17.782713 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.782676 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/799ed54bc75d16c836f178a66c89fbdc-config\") pod \"kube-apiserver-proxy-ip-10-0-131-147.us-west-2.compute.internal\" (UID: \"799ed54bc75d16c836f178a66c89fbdc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-147.us-west-2.compute.internal" Oct 30 19:24:17.782713 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.782703 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4cac457219e3215c4459c18e1cf7554f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-147.us-west-2.compute.internal\" (UID: \"4cac457219e3215c4459c18e1cf7554f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-147.us-west-2.compute.internal" Oct 30 19:24:17.782778 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.782718 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/799ed54bc75d16c836f178a66c89fbdc-config\") pod \"kube-apiserver-proxy-ip-10-0-131-147.us-west-2.compute.internal\" (UID: \"799ed54bc75d16c836f178a66c89fbdc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-147.us-west-2.compute.internal" Oct 30 19:24:17.782778 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.782745 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4cac457219e3215c4459c18e1cf7554f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-147.us-west-2.compute.internal\" (UID: \"4cac457219e3215c4459c18e1cf7554f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-147.us-west-2.compute.internal" Oct 30 19:24:17.842388 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.842340 2582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:24:17.842459 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.842446 2582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-147.us-west-2.compute.internal" Oct 30 19:24:17.899982 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.899960 2582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:24:17.947919 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.947895 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-147.us-west-2.compute.internal" Oct 30 19:24:17.948978 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:17.948954 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-147.us-west-2.compute.internal" Oct 30 19:24:18.323458 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.323223 2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 19:24:18.323996 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.323528 2582 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 19:24:18.323996 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.323551 2582 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 19:24:18.323996 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.323570 2582 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 19:24:18.323996 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:18.323612 2582 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a0c1794a904604bf0bf421b952c83549-4b7c0d71fd7f1dce.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-131-147.us-west-2.compute.internal.18e3651e4aa6588a\": read tcp 10.0.131.147:50076->18.227.202.105:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-131-147.us-west-2.compute.internal.18e3651e4aa6588a default 5032 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-147.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-147.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-147.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 19:24:17 +0000 UTC,LastTimestamp:2026-10-30 19:24:17.618020182 +0000 UTC m=+0.567199767,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-147.us-west-2.compute.internal,}" Oct 30 19:24:18.359389 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.359360 2582 apiserver.go:52] "Watching apiserver" Oct 30 19:24:18.379989 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.379968 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 19:24:18.594229 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.594146 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 19:19:17 +0000 UTC" deadline="2028-04-11 00:28:15.640922133 +0000 UTC" Oct 30 19:24:18.594229 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.594174 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12677h3m57.046750432s" Oct 30 19:24:18.667656 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.667629 2582 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 19:24:18.667982 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.667964 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-p6vhl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-147.us-west-2.compute.internal","kube-system/konnectivity-agent-l66t6","openshift-cluster-node-tuning-operator/tuned-672ql","openshift-network-diagnostics/network-check-target-t6m9p","kube-system/global-pull-secret-syncer-kcr6x","kube-system/kube-apiserver-proxy-ip-10-0-131-147.us-west-2.compute.internal","openshift-image-registry/node-ca-dqztf","openshift-multus/multus-additional-cni-plugins-x79ns","openshift-multus/multus-rffx6","openshift-ovn-kubernetes/ovnkube-node-phcbq","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqxgl","openshift-multus/network-metrics-daemon-mts99","openshift-network-operator/iptables-alerter-s5pd4"] Oct 30 19:24:18.670646 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.670631 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-p6vhl" Oct 30 19:24:18.672801 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.672781 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-l66t6" Oct 30 19:24:18.675016 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.675003 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:18.677248 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.677233 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t6m9p" Oct 30 19:24:18.677316 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:18.677297 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t6m9p" podUID="6ea94667-080f-4d20-887c-376aee648406" Oct 30 19:24:18.679391 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.679379 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kcr6x" Oct 30 19:24:18.679445 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:18.679427 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kcr6x" podUID="e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87" Oct 30 19:24:18.684089 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.684070 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dqztf" Oct 30 19:24:18.686152 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.686117 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-etc-modprobe-d\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:18.686269 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.686158 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-etc-sysctl-conf\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:18.686269 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.686226 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-x79ns" Oct 30 19:24:18.686269 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.686240 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1165f4d9-846b-4f12-92c1-944193f28d21-tmp\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:18.686462 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.686274 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a7d7990d-1d0f-41da-9dbf-1fbffe4af353-host\") pod \"node-ca-dqztf\" (UID: \"a7d7990d-1d0f-41da-9dbf-1fbffe4af353\") " pod="openshift-image-registry/node-ca-dqztf" Oct 30 19:24:18.686462 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.686304 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-etc-sysctl-d\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:18.686462 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.686345 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a7d7990d-1d0f-41da-9dbf-1fbffe4af353-serviceca\") pod \"node-ca-dqztf\" (UID: \"a7d7990d-1d0f-41da-9dbf-1fbffe4af353\") " pod="openshift-image-registry/node-ca-dqztf" Oct 30 19:24:18.686632 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.686471 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2198fe85-6a7d-4cb3-a9e1-f804851fdb1d-hosts-file\") pod \"node-resolver-p6vhl\" (UID: \"2198fe85-6a7d-4cb3-a9e1-f804851fdb1d\") " pod="openshift-dns/node-resolver-p6vhl" Oct 30 19:24:18.686632 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.686557 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-etc-kubernetes\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:18.686632 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.686595 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-var-lib-kubelet\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:18.686632 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.686620 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-etc-sysconfig\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:18.686815 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.686683 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-etc-systemd\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:18.686815 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.686711 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-sys\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:18.686815 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.686738 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nccr5\" (UniqueName: \"kubernetes.io/projected/1165f4d9-846b-4f12-92c1-944193f28d21-kube-api-access-nccr5\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:18.686815 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.686786 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87-dbus\") pod \"global-pull-secret-syncer-kcr6x\" (UID: \"e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87\") " pod="kube-system/global-pull-secret-syncer-kcr6x" Oct 30 19:24:18.686815 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.686808 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87-original-pull-secret\") pod \"global-pull-secret-syncer-kcr6x\" (UID: \"e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87\") " pod="kube-system/global-pull-secret-syncer-kcr6x" Oct 30 19:24:18.687282 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.686825 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtmm9\" (UniqueName: \"kubernetes.io/projected/2198fe85-6a7d-4cb3-a9e1-f804851fdb1d-kube-api-access-jtmm9\") pod \"node-resolver-p6vhl\" (UID: \"2198fe85-6a7d-4cb3-a9e1-f804851fdb1d\") " pod="openshift-dns/node-resolver-p6vhl" Oct 30 19:24:18.687282 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.686882 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-host\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:18.687282 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.686941 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj6h4\" (UniqueName: \"kubernetes.io/projected/6ea94667-080f-4d20-887c-376aee648406-kube-api-access-kj6h4\") pod \"network-check-target-t6m9p\" (UID: \"6ea94667-080f-4d20-887c-376aee648406\") " pod="openshift-network-diagnostics/network-check-target-t6m9p" Oct 30 19:24:18.687282 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.686977 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c9eecef1-2e58-4444-8a51-25b0b18715f5-agent-certs\") pod \"konnectivity-agent-l66t6\" (UID: \"c9eecef1-2e58-4444-8a51-25b0b18715f5\") " pod="kube-system/konnectivity-agent-l66t6" Oct 30 19:24:18.687282 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.687024 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1165f4d9-846b-4f12-92c1-944193f28d21-etc-tuned\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:18.687282 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.687052 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87-kubelet-config\") pod \"global-pull-secret-syncer-kcr6x\" (UID: \"e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87\") " pod="kube-system/global-pull-secret-syncer-kcr6x" Oct 30 19:24:18.687282 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.687080 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c9eecef1-2e58-4444-8a51-25b0b18715f5-konnectivity-ca\") pod \"konnectivity-agent-l66t6\" (UID: \"c9eecef1-2e58-4444-8a51-25b0b18715f5\") " pod="kube-system/konnectivity-agent-l66t6" Oct 30 19:24:18.687282 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.687142 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2198fe85-6a7d-4cb3-a9e1-f804851fdb1d-tmp-dir\") pod \"node-resolver-p6vhl\" (UID: \"2198fe85-6a7d-4cb3-a9e1-f804851fdb1d\") " pod="openshift-dns/node-resolver-p6vhl" Oct 30 19:24:18.687282 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.687181 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-run\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:18.687282 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.687226 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-lib-modules\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:18.687633 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.687288 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts8ds\" (UniqueName: \"kubernetes.io/projected/a7d7990d-1d0f-41da-9dbf-1fbffe4af353-kube-api-access-ts8ds\") pod \"node-ca-dqztf\" (UID: \"a7d7990d-1d0f-41da-9dbf-1fbffe4af353\") " pod="openshift-image-registry/node-ca-dqztf" Oct 30 19:24:18.688319 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.688302 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rffx6" Oct 30 19:24:18.690616 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.690603 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:18.692817 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.692798 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqxgl" Oct 30 19:24:18.695258 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.695234 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mts99" Oct 30 19:24:18.695332 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:18.695306 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mts99" podUID="adeacb06-6b16-4597-be99-ff84d8efdadc" Oct 30 19:24:18.697485 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.697464 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-s5pd4" Oct 30 19:24:18.729795 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.729777 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 19:24:18.729927 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.729912 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 19:24:18.730015 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.730001 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jxzqp\"" Oct 30 19:24:18.731061 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.731047 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 19:24:18.731490 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.731478 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 19:24:18.731583 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.731569 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-s4fj9\"" Oct 30 19:24:18.733224 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.733210 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:24:18.733273 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.733241 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 19:24:18.733404 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.733392 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hthrr\"" Oct 30 19:24:18.742848 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.742825 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 19:24:18.742958 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.742895 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 19:24:18.743035 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.743021 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 19:24:18.743125 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.743108 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-t2d46\"" Oct 30 19:24:18.744723 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.744709 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6prqp\"" Oct 30 19:24:18.745025 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.745012 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 19:24:18.745219 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.745203 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 19:24:18.745278 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.745236 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 19:24:18.745278 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.745248 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 19:24:18.745608 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.745593 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 19:24:18.781033 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.781016 2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 19:24:18.782242 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.782227 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 19:24:18.782334 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.782319 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 19:24:18.782534 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.782514 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-6ftg5\"" Oct 30 19:24:18.782609 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.782593 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 19:24:18.782659 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.782599 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 19:24:18.782659 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.782618 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6nsf6\"" Oct 30 19:24:18.782659 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.782637 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 19:24:18.782659 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.782641 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 19:24:18.783196 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.783183 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 19:24:18.785564 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.785545 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 19:24:18.785755 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.785737 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 19:24:18.785815 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.785751 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 19:24:18.785892 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.785880 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-ndzf5\"" Oct 30 19:24:18.786044 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.786032 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 19:24:18.786133 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.786118 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:24:18.786133 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.786127 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-lzgjh\"" Oct 30 19:24:18.786213 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.786175 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 19:24:18.788533 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.788515 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-etc-sysctl-d\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:18.788612 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.788548 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2198fe85-6a7d-4cb3-a9e1-f804851fdb1d-hosts-file\") pod \"node-resolver-p6vhl\" (UID: \"2198fe85-6a7d-4cb3-a9e1-f804851fdb1d\") " pod="openshift-dns/node-resolver-p6vhl" Oct 30 19:24:18.788612 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.788579 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-cni-binary-copy\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:18.788718 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.788673 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2198fe85-6a7d-4cb3-a9e1-f804851fdb1d-hosts-file\") pod \"node-resolver-p6vhl\" (UID: \"2198fe85-6a7d-4cb3-a9e1-f804851fdb1d\") " pod="openshift-dns/node-resolver-p6vhl" Oct 30 19:24:18.788718 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.788696 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-multus-conf-dir\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:18.788800 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.788720 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks5px\" (UniqueName: \"kubernetes.io/projected/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-kube-api-access-ks5px\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:18.788800 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.788742 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-etc-kubernetes\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:18.788800 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.788699 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-etc-sysctl-d\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:18.788800 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.788760 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-var-lib-kubelet\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:18.788918 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.788819 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-var-lib-kubelet\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:18.788918 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.788865 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-host-var-lib-cni-multus\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:18.788918 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.788881 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-etc-kubernetes\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:18.788918 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.788897 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-multus-daemon-config\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:18.788918 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.788916 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-host-run-netns\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:18.789072 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.788942 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:18.789072 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.788963 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-ovn-node-metrics-cert\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:18.789072 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.788980 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-etc-systemd\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:18.789072 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.789016 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-sys\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:18.789072 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.789031 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-etc-systemd\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:18.789072 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.789061 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nccr5\" (UniqueName: \"kubernetes.io/projected/1165f4d9-846b-4f12-92c1-944193f28d21-kube-api-access-nccr5\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:18.789368 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.789077 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87-original-pull-secret\") pod \"global-pull-secret-syncer-kcr6x\" (UID: \"e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87\") " pod="kube-system/global-pull-secret-syncer-kcr6x" Oct 30 19:24:18.789368 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.789119 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtmm9\" (UniqueName: \"kubernetes.io/projected/2198fe85-6a7d-4cb3-a9e1-f804851fdb1d-kube-api-access-jtmm9\") pod \"node-resolver-p6vhl\" (UID: \"2198fe85-6a7d-4cb3-a9e1-f804851fdb1d\") " pod="openshift-dns/node-resolver-p6vhl" Oct 30 19:24:18.789368 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:18.789155 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:24:18.789368 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.789223 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-host\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:18.789368 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:18.789249 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87-original-pull-secret podName:e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87 nodeName:}" failed. No retries permitted until 2026-10-30 19:24:19.28923151 +0000 UTC m=+2.238411103 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87-original-pull-secret") pod "global-pull-secret-syncer-kcr6x" (UID: "e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:24:18.789368 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.789247 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-sys\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:18.789368 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.789278 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-host\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:18.789368 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.789320 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj6h4\" (UniqueName: \"kubernetes.io/projected/6ea94667-080f-4d20-887c-376aee648406-kube-api-access-kj6h4\") pod \"network-check-target-t6m9p\" (UID: \"6ea94667-080f-4d20-887c-376aee648406\") " pod="openshift-network-diagnostics/network-check-target-t6m9p" Oct 30 19:24:18.789785 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.789380 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-cnibin\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:18.789785 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.789413 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-log-socket\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:18.789785 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.789454 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-host-run-ovn-kubernetes\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:18.789785 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.789488 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-ovnkube-config\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:18.789785 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.789516 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e1bb6034-1fd5-4a40-863d-e43314482f5c-socket-dir\") pod \"aws-ebs-csi-driver-node-rqxgl\" (UID: \"e1bb6034-1fd5-4a40-863d-e43314482f5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqxgl" Oct 30 19:24:18.789785 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.789566 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e1bb6034-1fd5-4a40-863d-e43314482f5c-device-dir\") pod \"aws-ebs-csi-driver-node-rqxgl\" (UID: \"e1bb6034-1fd5-4a40-863d-e43314482f5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqxgl" Oct 30 19:24:18.789785 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.789600 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87-kubelet-config\") pod \"global-pull-secret-syncer-kcr6x\" (UID: \"e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87\") " pod="kube-system/global-pull-secret-syncer-kcr6x" Oct 30 19:24:18.789785 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.789630 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-host-run-multus-certs\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:18.789785 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.789658 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klhnw\" (UniqueName: \"kubernetes.io/projected/adeacb06-6b16-4597-be99-ff84d8efdadc-kube-api-access-klhnw\") pod \"network-metrics-daemon-mts99\" (UID: \"adeacb06-6b16-4597-be99-ff84d8efdadc\") " pod="openshift-multus/network-metrics-daemon-mts99" Oct 30 19:24:18.789785 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.789660 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87-kubelet-config\") pod \"global-pull-secret-syncer-kcr6x\" (UID: \"e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87\") " pod="kube-system/global-pull-secret-syncer-kcr6x" Oct 30 19:24:18.789785 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.789685 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-hostroot\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:18.789785 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.789735 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-etc-kubernetes\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:18.789785 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.789763 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-node-log\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:18.789785 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.789787 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t95w\" (UniqueName: \"kubernetes.io/projected/fd873aa7-e90a-4819-8e78-ba123cd58cce-kube-api-access-8t95w\") pod \"iptables-alerter-s5pd4\" (UID: \"fd873aa7-e90a-4819-8e78-ba123cd58cce\") " pod="openshift-network-operator/iptables-alerter-s5pd4" Oct 30 19:24:18.790417 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.789830 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-systemd-units\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:18.790417 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.789856 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-etc-modprobe-d\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:18.790417 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.789916 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a7d7990d-1d0f-41da-9dbf-1fbffe4af353-host\") pod \"node-ca-dqztf\" (UID: \"a7d7990d-1d0f-41da-9dbf-1fbffe4af353\") " pod="openshift-image-registry/node-ca-dqztf" Oct 30 19:24:18.790417 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.789930 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-etc-modprobe-d\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:18.790417 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.789944 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-multus-socket-dir-parent\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:18.790417 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.789968 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-host-var-lib-cni-bin\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:18.790417 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.789974 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a7d7990d-1d0f-41da-9dbf-1fbffe4af353-host\") pod \"node-ca-dqztf\" (UID: \"a7d7990d-1d0f-41da-9dbf-1fbffe4af353\") " pod="openshift-image-registry/node-ca-dqztf" Oct 30 19:24:18.790417 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.790006 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-run-ovn\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:18.790417 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.790037 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68ccj\" (UniqueName: \"kubernetes.io/projected/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-kube-api-access-68ccj\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:18.790417 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.790059 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/83041ead-f80a-4e66-9ef2-44b25467952f-system-cni-dir\") pod \"multus-additional-cni-plugins-x79ns\" (UID: \"83041ead-f80a-4e66-9ef2-44b25467952f\") " pod="openshift-multus/multus-additional-cni-plugins-x79ns" Oct 30 19:24:18.790417 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.790076 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl5v5\" (UniqueName: \"kubernetes.io/projected/e1bb6034-1fd5-4a40-863d-e43314482f5c-kube-api-access-xl5v5\") pod \"aws-ebs-csi-driver-node-rqxgl\" (UID: \"e1bb6034-1fd5-4a40-863d-e43314482f5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqxgl" Oct 30 19:24:18.790417 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.790114 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a7d7990d-1d0f-41da-9dbf-1fbffe4af353-serviceca\") pod \"node-ca-dqztf\" (UID: \"a7d7990d-1d0f-41da-9dbf-1fbffe4af353\") " pod="openshift-image-registry/node-ca-dqztf" Oct 30 19:24:18.790417 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.790145 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-run-openvswitch\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:18.790417 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.790186 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-host-cni-netd\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:18.790417 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.790233 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-ovnkube-script-lib\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:18.790417 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.790258 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/83041ead-f80a-4e66-9ef2-44b25467952f-cni-binary-copy\") pod \"multus-additional-cni-plugins-x79ns\" (UID: \"83041ead-f80a-4e66-9ef2-44b25467952f\") " pod="openshift-multus/multus-additional-cni-plugins-x79ns" Oct 30 19:24:18.790417 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.790277 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e1bb6034-1fd5-4a40-863d-e43314482f5c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rqxgl\" (UID: \"e1bb6034-1fd5-4a40-863d-e43314482f5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqxgl" Oct 30 19:24:18.791115 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.790324 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e1bb6034-1fd5-4a40-863d-e43314482f5c-etc-selinux\") pod \"aws-ebs-csi-driver-node-rqxgl\" (UID: \"e1bb6034-1fd5-4a40-863d-e43314482f5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqxgl" Oct 30 19:24:18.791115 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.790375 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e1bb6034-1fd5-4a40-863d-e43314482f5c-sys-fs\") pod \"aws-ebs-csi-driver-node-rqxgl\" (UID: \"e1bb6034-1fd5-4a40-863d-e43314482f5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqxgl" Oct 30 19:24:18.791115 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.790393 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-host-run-netns\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:18.791115 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.790426 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-host-var-lib-kubelet\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:18.791115 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.790460 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-run-systemd\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:18.791115 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.790483 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/83041ead-f80a-4e66-9ef2-44b25467952f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x79ns\" (UID: \"83041ead-f80a-4e66-9ef2-44b25467952f\") " pod="openshift-multus/multus-additional-cni-plugins-x79ns" Oct 30 19:24:18.791115 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.790531 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/83041ead-f80a-4e66-9ef2-44b25467952f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x79ns\" (UID: \"83041ead-f80a-4e66-9ef2-44b25467952f\") " pod="openshift-multus/multus-additional-cni-plugins-x79ns" Oct 30 19:24:18.791115 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.790561 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fd873aa7-e90a-4819-8e78-ba123cd58cce-iptables-alerter-script\") pod \"iptables-alerter-s5pd4\" (UID: \"fd873aa7-e90a-4819-8e78-ba123cd58cce\") " pod="openshift-network-operator/iptables-alerter-s5pd4" Oct 30 19:24:18.791115 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.790583 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-etc-sysconfig\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:18.791115 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.790615 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87-dbus\") pod \"global-pull-secret-syncer-kcr6x\" (UID: \"e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87\") " pod="kube-system/global-pull-secret-syncer-kcr6x" Oct 30 19:24:18.791115 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.790624 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-etc-sysconfig\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:18.791115 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.790645 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a7d7990d-1d0f-41da-9dbf-1fbffe4af353-serviceca\") pod \"node-ca-dqztf\" (UID: \"a7d7990d-1d0f-41da-9dbf-1fbffe4af353\") " pod="openshift-image-registry/node-ca-dqztf" Oct 30 19:24:18.791115 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.790655 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-host-kubelet\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:18.791115 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.790673 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c9eecef1-2e58-4444-8a51-25b0b18715f5-agent-certs\") pod \"konnectivity-agent-l66t6\" (UID: \"c9eecef1-2e58-4444-8a51-25b0b18715f5\") " pod="kube-system/konnectivity-agent-l66t6" Oct 30 19:24:18.791115 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.790712 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-multus-cni-dir\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:18.791115 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.790734 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87-dbus\") pod \"global-pull-secret-syncer-kcr6x\" (UID: \"e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87\") " pod="kube-system/global-pull-secret-syncer-kcr6x" Oct 30 19:24:18.791115 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.790783 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbwtx\" (UniqueName: \"kubernetes.io/projected/83041ead-f80a-4e66-9ef2-44b25467952f-kube-api-access-jbwtx\") pod \"multus-additional-cni-plugins-x79ns\" (UID: \"83041ead-f80a-4e66-9ef2-44b25467952f\") " pod="openshift-multus/multus-additional-cni-plugins-x79ns" Oct 30 19:24:18.791889 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.790814 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e1bb6034-1fd5-4a40-863d-e43314482f5c-registration-dir\") pod \"aws-ebs-csi-driver-node-rqxgl\" (UID: \"e1bb6034-1fd5-4a40-863d-e43314482f5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqxgl" Oct 30 19:24:18.791889 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.790833 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fd873aa7-e90a-4819-8e78-ba123cd58cce-host-slash\") pod \"iptables-alerter-s5pd4\" (UID: \"fd873aa7-e90a-4819-8e78-ba123cd58cce\") " pod="openshift-network-operator/iptables-alerter-s5pd4" Oct 30 19:24:18.791889 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.790875 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1165f4d9-846b-4f12-92c1-944193f28d21-etc-tuned\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:18.791889 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.790906 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c9eecef1-2e58-4444-8a51-25b0b18715f5-konnectivity-ca\") pod \"konnectivity-agent-l66t6\" (UID: \"c9eecef1-2e58-4444-8a51-25b0b18715f5\") " pod="kube-system/konnectivity-agent-l66t6" Oct 30 19:24:18.791889 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.790941 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-system-cni-dir\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:18.791889 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.791009 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/83041ead-f80a-4e66-9ef2-44b25467952f-cnibin\") pod \"multus-additional-cni-plugins-x79ns\" (UID: \"83041ead-f80a-4e66-9ef2-44b25467952f\") " pod="openshift-multus/multus-additional-cni-plugins-x79ns" Oct 30 19:24:18.791889 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.791022 2582 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 19:24:18.791889 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.791054 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/83041ead-f80a-4e66-9ef2-44b25467952f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x79ns\" (UID: \"83041ead-f80a-4e66-9ef2-44b25467952f\") " pod="openshift-multus/multus-additional-cni-plugins-x79ns" Oct 30 19:24:18.791889 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.791096 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adeacb06-6b16-4597-be99-ff84d8efdadc-metrics-certs\") pod \"network-metrics-daemon-mts99\" (UID: \"adeacb06-6b16-4597-be99-ff84d8efdadc\") " pod="openshift-multus/network-metrics-daemon-mts99" Oct 30 19:24:18.791889 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.791133 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2198fe85-6a7d-4cb3-a9e1-f804851fdb1d-tmp-dir\") pod \"node-resolver-p6vhl\" (UID: \"2198fe85-6a7d-4cb3-a9e1-f804851fdb1d\") " pod="openshift-dns/node-resolver-p6vhl" Oct 30 19:24:18.791889 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.791177 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-host-run-k8s-cni-cncf-io\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:18.791889 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.791210 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-run\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:18.791889 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.791234 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-lib-modules\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:18.791889 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.791256 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts8ds\" (UniqueName: \"kubernetes.io/projected/a7d7990d-1d0f-41da-9dbf-1fbffe4af353-kube-api-access-ts8ds\") pod \"node-ca-dqztf\" (UID: \"a7d7990d-1d0f-41da-9dbf-1fbffe4af353\") " pod="openshift-image-registry/node-ca-dqztf" Oct 30 19:24:18.791889 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.791279 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-host-slash\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:18.791889 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.791296 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-etc-openvswitch\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:18.791889 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.791300 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-run\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:18.792757 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.791312 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-host-cni-bin\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:18.792757 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.791335 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-env-overrides\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:18.792757 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.791379 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-lib-modules\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:18.792757 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.791414 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-etc-sysctl-conf\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:18.792757 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.791452 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1165f4d9-846b-4f12-92c1-944193f28d21-tmp\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:18.792757 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.791471 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c9eecef1-2e58-4444-8a51-25b0b18715f5-konnectivity-ca\") pod \"konnectivity-agent-l66t6\" (UID: \"c9eecef1-2e58-4444-8a51-25b0b18715f5\") " pod="kube-system/konnectivity-agent-l66t6" Oct 30 19:24:18.792757 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.791474 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2198fe85-6a7d-4cb3-a9e1-f804851fdb1d-tmp-dir\") pod \"node-resolver-p6vhl\" (UID: \"2198fe85-6a7d-4cb3-a9e1-f804851fdb1d\") " pod="openshift-dns/node-resolver-p6vhl" Oct 30 19:24:18.792757 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.791528 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-var-lib-openvswitch\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:18.792757 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.791547 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-etc-sysctl-conf\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:18.794824 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.794797 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1165f4d9-846b-4f12-92c1-944193f28d21-etc-tuned\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:18.794912 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.794826 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1165f4d9-846b-4f12-92c1-944193f28d21-tmp\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:18.794951 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.794921 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c9eecef1-2e58-4444-8a51-25b0b18715f5-agent-certs\") pod \"konnectivity-agent-l66t6\" (UID: \"c9eecef1-2e58-4444-8a51-25b0b18715f5\") " pod="kube-system/konnectivity-agent-l66t6" Oct 30 19:24:18.836160 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.836137 2582 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 19:24:18.836278 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.836194 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-84cmj" Oct 30 19:24:18.854862 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:18.854804 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:24:18.854862 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:18.854827 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:24:18.854862 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:18.854840 2582 projected.go:196] Error preparing data for projected volume kube-api-access-kj6h4 for pod openshift-network-diagnostics/network-check-target-t6m9p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:24:18.855065 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:18.854910 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ea94667-080f-4d20-887c-376aee648406-kube-api-access-kj6h4 podName:6ea94667-080f-4d20-887c-376aee648406 nodeName:}" failed. No retries permitted until 2026-10-30 19:24:19.354895386 +0000 UTC m=+2.304074956 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-kj6h4" (UniqueName: "kubernetes.io/projected/6ea94667-080f-4d20-887c-376aee648406-kube-api-access-kj6h4") pod "network-check-target-t6m9p" (UID: "6ea94667-080f-4d20-887c-376aee648406") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:24:18.857040 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.857011 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nccr5\" (UniqueName: \"kubernetes.io/projected/1165f4d9-846b-4f12-92c1-944193f28d21-kube-api-access-nccr5\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:18.857184 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.857164 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtmm9\" (UniqueName: \"kubernetes.io/projected/2198fe85-6a7d-4cb3-a9e1-f804851fdb1d-kube-api-access-jtmm9\") pod \"node-resolver-p6vhl\" (UID: \"2198fe85-6a7d-4cb3-a9e1-f804851fdb1d\") " pod="openshift-dns/node-resolver-p6vhl" Oct 30 19:24:18.858640 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.858614 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts8ds\" (UniqueName: \"kubernetes.io/projected/a7d7990d-1d0f-41da-9dbf-1fbffe4af353-kube-api-access-ts8ds\") pod \"node-ca-dqztf\" (UID: \"a7d7990d-1d0f-41da-9dbf-1fbffe4af353\") " pod="openshift-image-registry/node-ca-dqztf" Oct 30 19:24:18.892169 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.892131 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-host-run-k8s-cni-cncf-io\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:18.892306 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.892180 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-host-slash\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:18.892306 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.892204 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-etc-openvswitch\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:18.892306 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.892232 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-host-cni-bin\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:18.892306 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.892253 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-host-run-k8s-cni-cncf-io\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:18.892306 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.892258 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-host-slash\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:18.892306 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.892282 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-env-overrides\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:18.892626 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.892311 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-etc-openvswitch\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:18.892626 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.892312 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-var-lib-openvswitch\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:18.892626 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.892351 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-cni-binary-copy\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:18.892626 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.892338 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-host-cni-bin\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:18.892626 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.892353 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-var-lib-openvswitch\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:18.892626 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.892399 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-multus-conf-dir\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:18.892626 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.892445 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks5px\" (UniqueName: \"kubernetes.io/projected/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-kube-api-access-ks5px\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:18.892626 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.892475 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-host-var-lib-cni-multus\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:18.892626 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.892484 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-multus-conf-dir\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:18.892626 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.892499 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-multus-daemon-config\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:18.892626 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.892525 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-host-run-netns\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:18.892626 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.892556 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:18.892626 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.892584 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-ovn-node-metrics-cert\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:18.892626 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.892555 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-host-var-lib-cni-multus\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:18.893334 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.892650 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:18.893334 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.892666 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-cnibin\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:18.893334 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.892695 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-log-socket\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:18.893334 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.892707 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-host-run-netns\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:18.893334 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.892724 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-host-run-ovn-kubernetes\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:18.893334 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.892749 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-ovnkube-config\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:18.893334 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.892775 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e1bb6034-1fd5-4a40-863d-e43314482f5c-socket-dir\") pod \"aws-ebs-csi-driver-node-rqxgl\" (UID: \"e1bb6034-1fd5-4a40-863d-e43314482f5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqxgl" Oct 30 19:24:18.893334 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.892801 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e1bb6034-1fd5-4a40-863d-e43314482f5c-device-dir\") pod \"aws-ebs-csi-driver-node-rqxgl\" (UID: \"e1bb6034-1fd5-4a40-863d-e43314482f5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqxgl" Oct 30 19:24:18.893334 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.892819 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-host-run-ovn-kubernetes\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:18.893334 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.892827 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-host-run-multus-certs\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:18.893334 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.892857 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-env-overrides\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:18.893334 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.892866 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klhnw\" (UniqueName: \"kubernetes.io/projected/adeacb06-6b16-4597-be99-ff84d8efdadc-kube-api-access-klhnw\") pod \"network-metrics-daemon-mts99\" (UID: \"adeacb06-6b16-4597-be99-ff84d8efdadc\") " pod="openshift-multus/network-metrics-daemon-mts99" Oct 30 19:24:18.893334 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.892864 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-host-run-multus-certs\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:18.893334 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.892896 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-hostroot\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:18.893334 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.892923 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-etc-kubernetes\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:18.893334 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.892934 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-log-socket\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:18.893334 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.892950 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-node-log\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:18.894250 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.892965 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e1bb6034-1fd5-4a40-863d-e43314482f5c-socket-dir\") pod \"aws-ebs-csi-driver-node-rqxgl\" (UID: \"e1bb6034-1fd5-4a40-863d-e43314482f5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqxgl" Oct 30 19:24:18.894250 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.892978 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t95w\" (UniqueName: \"kubernetes.io/projected/fd873aa7-e90a-4819-8e78-ba123cd58cce-kube-api-access-8t95w\") pod \"iptables-alerter-s5pd4\" (UID: \"fd873aa7-e90a-4819-8e78-ba123cd58cce\") " pod="openshift-network-operator/iptables-alerter-s5pd4" Oct 30 19:24:18.894250 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.892776 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-cnibin\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:18.894250 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.892983 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e1bb6034-1fd5-4a40-863d-e43314482f5c-device-dir\") pod \"aws-ebs-csi-driver-node-rqxgl\" (UID: \"e1bb6034-1fd5-4a40-863d-e43314482f5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqxgl" Oct 30 19:24:18.894250 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.893004 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-systemd-units\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:18.894250 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.893030 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-hostroot\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:18.894250 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.893033 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-multus-socket-dir-parent\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:18.894250 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.893088 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-host-var-lib-cni-bin\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:18.894250 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.893113 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-run-ovn\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:18.894250 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.893128 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-cni-binary-copy\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:18.894250 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.893139 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68ccj\" (UniqueName: \"kubernetes.io/projected/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-kube-api-access-68ccj\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:18.894250 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.893144 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-multus-daemon-config\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:18.894250 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.893168 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/83041ead-f80a-4e66-9ef2-44b25467952f-system-cni-dir\") pod \"multus-additional-cni-plugins-x79ns\" (UID: \"83041ead-f80a-4e66-9ef2-44b25467952f\") " pod="openshift-multus/multus-additional-cni-plugins-x79ns" Oct 30 19:24:18.894250 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.893205 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-node-log\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:18.894250 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.893215 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/83041ead-f80a-4e66-9ef2-44b25467952f-system-cni-dir\") pod \"multus-additional-cni-plugins-x79ns\" (UID: \"83041ead-f80a-4e66-9ef2-44b25467952f\") " pod="openshift-multus/multus-additional-cni-plugins-x79ns" Oct 30 19:24:18.894250 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.893206 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl5v5\" (UniqueName: \"kubernetes.io/projected/e1bb6034-1fd5-4a40-863d-e43314482f5c-kube-api-access-xl5v5\") pod \"aws-ebs-csi-driver-node-rqxgl\" (UID: \"e1bb6034-1fd5-4a40-863d-e43314482f5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqxgl" Oct 30 19:24:18.894250 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.893256 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-run-openvswitch\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:18.894938 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.893264 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-host-var-lib-cni-bin\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:18.894938 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.893275 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-multus-socket-dir-parent\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:18.894938 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.893288 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-host-cni-netd\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:18.894938 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.893296 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-ovnkube-config\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:18.894938 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.893302 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-run-ovn\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:18.894938 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.893317 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-ovnkube-script-lib\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:18.894938 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.893317 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-etc-kubernetes\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:18.894938 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.893346 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/83041ead-f80a-4e66-9ef2-44b25467952f-cni-binary-copy\") pod \"multus-additional-cni-plugins-x79ns\" (UID: \"83041ead-f80a-4e66-9ef2-44b25467952f\") " pod="openshift-multus/multus-additional-cni-plugins-x79ns" Oct 30 19:24:18.894938 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.893354 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-systemd-units\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:18.894938 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.893369 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e1bb6034-1fd5-4a40-863d-e43314482f5c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rqxgl\" (UID: \"e1bb6034-1fd5-4a40-863d-e43314482f5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqxgl" Oct 30 19:24:18.894938 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.893371 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-host-cni-netd\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:18.894938 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.893394 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e1bb6034-1fd5-4a40-863d-e43314482f5c-etc-selinux\") pod \"aws-ebs-csi-driver-node-rqxgl\" (UID: \"e1bb6034-1fd5-4a40-863d-e43314482f5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqxgl" Oct 30 19:24:18.894938 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.893435 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e1bb6034-1fd5-4a40-863d-e43314482f5c-sys-fs\") pod \"aws-ebs-csi-driver-node-rqxgl\" (UID: \"e1bb6034-1fd5-4a40-863d-e43314482f5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqxgl" Oct 30 19:24:18.894938 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.893460 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-host-run-netns\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:18.894938 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.893472 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-run-openvswitch\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:18.894938 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.893489 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-host-var-lib-kubelet\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:18.894938 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.893506 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e1bb6034-1fd5-4a40-863d-e43314482f5c-etc-selinux\") pod \"aws-ebs-csi-driver-node-rqxgl\" (UID: \"e1bb6034-1fd5-4a40-863d-e43314482f5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqxgl" Oct 30 19:24:18.895479 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.893514 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-run-systemd\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:18.895479 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.893531 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e1bb6034-1fd5-4a40-863d-e43314482f5c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rqxgl\" (UID: \"e1bb6034-1fd5-4a40-863d-e43314482f5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqxgl" Oct 30 19:24:18.895479 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.893550 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/83041ead-f80a-4e66-9ef2-44b25467952f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x79ns\" (UID: \"83041ead-f80a-4e66-9ef2-44b25467952f\") " pod="openshift-multus/multus-additional-cni-plugins-x79ns" Oct 30 19:24:18.895479 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.893552 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-host-run-netns\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:18.895479 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.893556 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-run-systemd\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:18.895479 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.893556 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-host-var-lib-kubelet\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:18.895479 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.893597 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/83041ead-f80a-4e66-9ef2-44b25467952f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x79ns\" (UID: \"83041ead-f80a-4e66-9ef2-44b25467952f\") " pod="openshift-multus/multus-additional-cni-plugins-x79ns" Oct 30 19:24:18.895479 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.893631 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fd873aa7-e90a-4819-8e78-ba123cd58cce-iptables-alerter-script\") pod \"iptables-alerter-s5pd4\" (UID: \"fd873aa7-e90a-4819-8e78-ba123cd58cce\") " pod="openshift-network-operator/iptables-alerter-s5pd4" Oct 30 19:24:18.895479 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.893662 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-host-kubelet\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:18.895479 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.893724 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-multus-cni-dir\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:18.895479 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.893808 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbwtx\" (UniqueName: \"kubernetes.io/projected/83041ead-f80a-4e66-9ef2-44b25467952f-kube-api-access-jbwtx\") pod \"multus-additional-cni-plugins-x79ns\" (UID: \"83041ead-f80a-4e66-9ef2-44b25467952f\") " pod="openshift-multus/multus-additional-cni-plugins-x79ns" Oct 30 19:24:18.895479 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.893836 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e1bb6034-1fd5-4a40-863d-e43314482f5c-registration-dir\") pod \"aws-ebs-csi-driver-node-rqxgl\" (UID: \"e1bb6034-1fd5-4a40-863d-e43314482f5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqxgl" Oct 30 19:24:18.895479 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.893864 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fd873aa7-e90a-4819-8e78-ba123cd58cce-host-slash\") pod \"iptables-alerter-s5pd4\" (UID: \"fd873aa7-e90a-4819-8e78-ba123cd58cce\") " pod="openshift-network-operator/iptables-alerter-s5pd4" Oct 30 19:24:18.895479 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.893890 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-multus-cni-dir\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:18.895479 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.893738 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/83041ead-f80a-4e66-9ef2-44b25467952f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x79ns\" (UID: \"83041ead-f80a-4e66-9ef2-44b25467952f\") " pod="openshift-multus/multus-additional-cni-plugins-x79ns" Oct 30 19:24:18.895479 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.893947 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-system-cni-dir\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:18.895479 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.893891 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-system-cni-dir\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:18.896016 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.893960 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-ovnkube-script-lib\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:18.896016 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.893987 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/83041ead-f80a-4e66-9ef2-44b25467952f-cnibin\") pod \"multus-additional-cni-plugins-x79ns\" (UID: \"83041ead-f80a-4e66-9ef2-44b25467952f\") " pod="openshift-multus/multus-additional-cni-plugins-x79ns" Oct 30 19:24:18.896016 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.894015 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/83041ead-f80a-4e66-9ef2-44b25467952f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x79ns\" (UID: \"83041ead-f80a-4e66-9ef2-44b25467952f\") " pod="openshift-multus/multus-additional-cni-plugins-x79ns" Oct 30 19:24:18.896016 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.894029 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e1bb6034-1fd5-4a40-863d-e43314482f5c-registration-dir\") pod \"aws-ebs-csi-driver-node-rqxgl\" (UID: \"e1bb6034-1fd5-4a40-863d-e43314482f5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqxgl" Oct 30 19:24:18.896016 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.893774 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-host-kubelet\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:18.896016 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.894044 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adeacb06-6b16-4597-be99-ff84d8efdadc-metrics-certs\") pod \"network-metrics-daemon-mts99\" (UID: \"adeacb06-6b16-4597-be99-ff84d8efdadc\") " pod="openshift-multus/network-metrics-daemon-mts99" Oct 30 19:24:18.896016 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.893601 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e1bb6034-1fd5-4a40-863d-e43314482f5c-sys-fs\") pod \"aws-ebs-csi-driver-node-rqxgl\" (UID: \"e1bb6034-1fd5-4a40-863d-e43314482f5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqxgl" Oct 30 19:24:18.896016 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:18.894115 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:24:18.896016 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.894127 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fd873aa7-e90a-4819-8e78-ba123cd58cce-host-slash\") pod \"iptables-alerter-s5pd4\" (UID: \"fd873aa7-e90a-4819-8e78-ba123cd58cce\") " pod="openshift-network-operator/iptables-alerter-s5pd4" Oct 30 19:24:18.896016 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:18.894165 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/adeacb06-6b16-4597-be99-ff84d8efdadc-metrics-certs podName:adeacb06-6b16-4597-be99-ff84d8efdadc nodeName:}" failed. No retries permitted until 2026-10-30 19:24:19.394149763 +0000 UTC m=+2.343329340 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/adeacb06-6b16-4597-be99-ff84d8efdadc-metrics-certs") pod "network-metrics-daemon-mts99" (UID: "adeacb06-6b16-4597-be99-ff84d8efdadc") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:24:18.896016 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.894167 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/83041ead-f80a-4e66-9ef2-44b25467952f-cni-binary-copy\") pod \"multus-additional-cni-plugins-x79ns\" (UID: \"83041ead-f80a-4e66-9ef2-44b25467952f\") " pod="openshift-multus/multus-additional-cni-plugins-x79ns" Oct 30 19:24:18.896016 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.894212 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/83041ead-f80a-4e66-9ef2-44b25467952f-cnibin\") pod \"multus-additional-cni-plugins-x79ns\" (UID: \"83041ead-f80a-4e66-9ef2-44b25467952f\") " pod="openshift-multus/multus-additional-cni-plugins-x79ns" Oct 30 19:24:18.896016 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.894284 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fd873aa7-e90a-4819-8e78-ba123cd58cce-iptables-alerter-script\") pod \"iptables-alerter-s5pd4\" (UID: \"fd873aa7-e90a-4819-8e78-ba123cd58cce\") " pod="openshift-network-operator/iptables-alerter-s5pd4" Oct 30 19:24:18.896016 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.894698 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/83041ead-f80a-4e66-9ef2-44b25467952f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x79ns\" (UID: \"83041ead-f80a-4e66-9ef2-44b25467952f\") " pod="openshift-multus/multus-additional-cni-plugins-x79ns" Oct 30 19:24:18.896016 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.894932 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/83041ead-f80a-4e66-9ef2-44b25467952f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x79ns\" (UID: \"83041ead-f80a-4e66-9ef2-44b25467952f\") " pod="openshift-multus/multus-additional-cni-plugins-x79ns" Oct 30 19:24:18.896016 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.895157 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-ovn-node-metrics-cert\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:18.962578 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.962539 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks5px\" (UniqueName: \"kubernetes.io/projected/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-kube-api-access-ks5px\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:18.963496 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.963446 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl5v5\" (UniqueName: \"kubernetes.io/projected/e1bb6034-1fd5-4a40-863d-e43314482f5c-kube-api-access-xl5v5\") pod \"aws-ebs-csi-driver-node-rqxgl\" (UID: \"e1bb6034-1fd5-4a40-863d-e43314482f5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqxgl" Oct 30 19:24:18.963665 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.963639 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klhnw\" (UniqueName: \"kubernetes.io/projected/adeacb06-6b16-4597-be99-ff84d8efdadc-kube-api-access-klhnw\") pod \"network-metrics-daemon-mts99\" (UID: \"adeacb06-6b16-4597-be99-ff84d8efdadc\") " pod="openshift-multus/network-metrics-daemon-mts99" Oct 30 19:24:18.963841 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.963812 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbwtx\" (UniqueName: \"kubernetes.io/projected/83041ead-f80a-4e66-9ef2-44b25467952f-kube-api-access-jbwtx\") pod \"multus-additional-cni-plugins-x79ns\" (UID: \"83041ead-f80a-4e66-9ef2-44b25467952f\") " pod="openshift-multus/multus-additional-cni-plugins-x79ns" Oct 30 19:24:18.963951 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.963928 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t95w\" (UniqueName: \"kubernetes.io/projected/fd873aa7-e90a-4819-8e78-ba123cd58cce-kube-api-access-8t95w\") pod \"iptables-alerter-s5pd4\" (UID: \"fd873aa7-e90a-4819-8e78-ba123cd58cce\") " pod="openshift-network-operator/iptables-alerter-s5pd4" Oct 30 19:24:18.964152 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:18.964136 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68ccj\" (UniqueName: \"kubernetes.io/projected/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-kube-api-access-68ccj\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:19.004673 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:19.004649 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-p6vhl" Oct 30 19:24:19.012635 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:19.012608 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-l66t6" Oct 30 19:24:19.028598 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:19.028570 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:19.041965 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:19.041912 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2198fe85_6a7d_4cb3_a9e1_f804851fdb1d.slice/crio-11f7ed69235aad4fde75385d30e20e7bcddeb953f8af831aaa10956067e145f3 WatchSource:0}: Error finding container 11f7ed69235aad4fde75385d30e20e7bcddeb953f8af831aaa10956067e145f3: Status 404 returned error can't find the container with id 11f7ed69235aad4fde75385d30e20e7bcddeb953f8af831aaa10956067e145f3 Oct 30 19:24:19.042554 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:19.042527 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9eecef1_2e58_4444_8a51_25b0b18715f5.slice/crio-fb567e9b5fd55e8e41d6248cb814e780ec6e5c2490b72fadbbbdcaa8b2a8de4c WatchSource:0}: Error finding container fb567e9b5fd55e8e41d6248cb814e780ec6e5c2490b72fadbbbdcaa8b2a8de4c: Status 404 returned error can't find the container with id fb567e9b5fd55e8e41d6248cb814e780ec6e5c2490b72fadbbbdcaa8b2a8de4c Oct 30 19:24:19.043345 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:19.043322 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cac457219e3215c4459c18e1cf7554f.slice/crio-f04dfc935e4256d3d2231f330ea198145045d2c6f31ece3289f9f72fbc680d4f WatchSource:0}: Error finding container f04dfc935e4256d3d2231f330ea198145045d2c6f31ece3289f9f72fbc680d4f: Status 404 returned error can't find the container with id f04dfc935e4256d3d2231f330ea198145045d2c6f31ece3289f9f72fbc680d4f Oct 30 19:24:19.043947 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:19.043914 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod799ed54bc75d16c836f178a66c89fbdc.slice/crio-b54d0851972c77e8e6a51e74806c7af996506eef1df90a95cd4627860359c4a1 WatchSource:0}: Error finding container b54d0851972c77e8e6a51e74806c7af996506eef1df90a95cd4627860359c4a1: Status 404 returned error can't find the container with id b54d0851972c77e8e6a51e74806c7af996506eef1df90a95cd4627860359c4a1 Oct 30 19:24:19.045042 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:19.045019 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dqztf" Oct 30 19:24:19.046036 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:19.046015 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1165f4d9_846b_4f12_92c1_944193f28d21.slice/crio-0489dbb7ced385b338ddf87bce1d5b28a4ab00f111c79b47e1778a31b8d58b55 WatchSource:0}: Error finding container 0489dbb7ced385b338ddf87bce1d5b28a4ab00f111c79b47e1778a31b8d58b55: Status 404 returned error can't find the container with id 0489dbb7ced385b338ddf87bce1d5b28a4ab00f111c79b47e1778a31b8d58b55 Oct 30 19:24:19.048651 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:19.048595 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:24:19.055075 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:19.055056 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7d7990d_1d0f_41da_9dbf_1fbffe4af353.slice/crio-8abb9cdd72c205c969b92f61e96d8a8f8ff42513f80a6718bdb6ccf464ba5f6b WatchSource:0}: Error finding container 8abb9cdd72c205c969b92f61e96d8a8f8ff42513f80a6718bdb6ccf464ba5f6b: Status 404 returned error can't find the container with id 8abb9cdd72c205c969b92f61e96d8a8f8ff42513f80a6718bdb6ccf464ba5f6b Oct 30 19:24:19.071275 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:19.071254 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-x79ns" Oct 30 19:24:19.073533 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:19.073510 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rffx6" Oct 30 19:24:19.081242 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:19.081216 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83041ead_f80a_4e66_9ef2_44b25467952f.slice/crio-ee9b88444d4f93ce2440c6b6ba3527ca9e6620a8bc1d508110b422462067d96f WatchSource:0}: Error finding container ee9b88444d4f93ce2440c6b6ba3527ca9e6620a8bc1d508110b422462067d96f: Status 404 returned error can't find the container with id ee9b88444d4f93ce2440c6b6ba3527ca9e6620a8bc1d508110b422462067d96f Oct 30 19:24:19.084011 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:19.083986 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5e15534_ee0e_444b_b3ac_4a5f21c1fcc1.slice/crio-413c7b1cf9af624b23d5f9434258acc869486b4d718e815b2888be6ff87c381f WatchSource:0}: Error finding container 413c7b1cf9af624b23d5f9434258acc869486b4d718e815b2888be6ff87c381f: Status 404 returned error can't find the container with id 413c7b1cf9af624b23d5f9434258acc869486b4d718e815b2888be6ff87c381f Oct 30 19:24:19.116187 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:19.116143 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:19.117279 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:19.117263 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-s5pd4" Oct 30 19:24:19.119614 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:19.119599 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqxgl" Oct 30 19:24:19.126307 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:19.126283 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb4226c2_13b5_4ef8_97ba_ffd6126442f4.slice/crio-3cc73881ed59439fc78eb71b370e6d849851e175732e0f2187be3d1d42adaef6 WatchSource:0}: Error finding container 3cc73881ed59439fc78eb71b370e6d849851e175732e0f2187be3d1d42adaef6: Status 404 returned error can't find the container with id 3cc73881ed59439fc78eb71b370e6d849851e175732e0f2187be3d1d42adaef6 Oct 30 19:24:19.132142 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:19.132116 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd873aa7_e90a_4819_8e78_ba123cd58cce.slice/crio-315a0f7a132b2ab04e7ac75fe697915b0a98ff6d3fdf50001fbcd4d4adaf5d2a WatchSource:0}: Error finding container 315a0f7a132b2ab04e7ac75fe697915b0a98ff6d3fdf50001fbcd4d4adaf5d2a: Status 404 returned error can't find the container with id 315a0f7a132b2ab04e7ac75fe697915b0a98ff6d3fdf50001fbcd4d4adaf5d2a Oct 30 19:24:19.134885 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:19.134862 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1bb6034_1fd5_4a40_863d_e43314482f5c.slice/crio-3ff80dae5315e666f230525f1bcd98c989b845b44a07096c0563a1681be3e567 WatchSource:0}: Error finding container 3ff80dae5315e666f230525f1bcd98c989b845b44a07096c0563a1681be3e567: Status 404 returned error can't find the container with id 3ff80dae5315e666f230525f1bcd98c989b845b44a07096c0563a1681be3e567 Oct 30 19:24:19.136702 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:19.136683 2582 kuberuntime_manager.go:1707] "Unhandled Error" err="container csi-driver start failed in pod aws-ebs-csi-driver-node-rqxgl_openshift-cluster-csi-drivers(e1bb6034-1fd5-4a40-863d-e43314482f5c): ErrImagePull: pull QPS exceeded" logger="UnhandledError" Oct 30 19:24:19.137639 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:19.137621 2582 kuberuntime_manager.go:1707] "Unhandled Error" err="container csi-node-driver-registrar start failed in pod aws-ebs-csi-driver-node-rqxgl_openshift-cluster-csi-drivers(e1bb6034-1fd5-4a40-863d-e43314482f5c): ErrImagePull: pull QPS exceeded" logger="UnhandledError" Oct 30 19:24:19.138830 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:19.138813 2582 kuberuntime_manager.go:1707] "Unhandled Error" err="container csi-liveness-probe start failed in pod aws-ebs-csi-driver-node-rqxgl_openshift-cluster-csi-drivers(e1bb6034-1fd5-4a40-863d-e43314482f5c): ErrImagePull: pull QPS exceeded" logger="UnhandledError" Oct 30 19:24:19.138886 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:19.138848 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-driver\" with ErrImagePull: \"pull QPS exceeded\", failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ErrImagePull: \"pull QPS exceeded\", failed to \"StartContainer\" for \"csi-liveness-probe\" with ErrImagePull: \"pull QPS exceeded\"]" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqxgl" podUID="e1bb6034-1fd5-4a40-863d-e43314482f5c" Oct 30 19:24:19.296866 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:19.296831 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87-original-pull-secret\") pod \"global-pull-secret-syncer-kcr6x\" (UID: \"e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87\") " pod="kube-system/global-pull-secret-syncer-kcr6x" Oct 30 19:24:19.297022 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:19.296985 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:24:19.297066 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:19.297045 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87-original-pull-secret podName:e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87 nodeName:}" failed. No retries permitted until 2026-10-30 19:24:20.297030895 +0000 UTC m=+3.246210483 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87-original-pull-secret") pod "global-pull-secret-syncer-kcr6x" (UID: "e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:24:19.397533 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:19.397454 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adeacb06-6b16-4597-be99-ff84d8efdadc-metrics-certs\") pod \"network-metrics-daemon-mts99\" (UID: \"adeacb06-6b16-4597-be99-ff84d8efdadc\") " pod="openshift-multus/network-metrics-daemon-mts99" Oct 30 19:24:19.397972 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:19.397534 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj6h4\" (UniqueName: \"kubernetes.io/projected/6ea94667-080f-4d20-887c-376aee648406-kube-api-access-kj6h4\") pod \"network-check-target-t6m9p\" (UID: \"6ea94667-080f-4d20-887c-376aee648406\") " pod="openshift-network-diagnostics/network-check-target-t6m9p" Oct 30 19:24:19.397972 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:19.397574 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:24:19.397972 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:19.397639 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:24:19.397972 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:19.397656 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:24:19.397972 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:19.397669 2582 projected.go:196] Error preparing data for projected volume kube-api-access-kj6h4 for pod openshift-network-diagnostics/network-check-target-t6m9p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:24:19.397972 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:19.397641 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/adeacb06-6b16-4597-be99-ff84d8efdadc-metrics-certs podName:adeacb06-6b16-4597-be99-ff84d8efdadc nodeName:}" failed. No retries permitted until 2026-10-30 19:24:20.397626142 +0000 UTC m=+3.346805711 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/adeacb06-6b16-4597-be99-ff84d8efdadc-metrics-certs") pod "network-metrics-daemon-mts99" (UID: "adeacb06-6b16-4597-be99-ff84d8efdadc") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:24:19.397972 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:19.397716 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ea94667-080f-4d20-887c-376aee648406-kube-api-access-kj6h4 podName:6ea94667-080f-4d20-887c-376aee648406 nodeName:}" failed. No retries permitted until 2026-10-30 19:24:20.397704305 +0000 UTC m=+3.346883874 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-kj6h4" (UniqueName: "kubernetes.io/projected/6ea94667-080f-4d20-887c-376aee648406-kube-api-access-kj6h4") pod "network-check-target-t6m9p" (UID: "6ea94667-080f-4d20-887c-376aee648406") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:24:19.414918 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:19.414893 2582 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:24:19.521397 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:19.521340 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqxgl" event={"ID":"e1bb6034-1fd5-4a40-863d-e43314482f5c","Type":"ContainerStarted","Data":"3ff80dae5315e666f230525f1bcd98c989b845b44a07096c0563a1681be3e567"} Oct 30 19:24:19.522596 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:19.522574 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-s5pd4" event={"ID":"fd873aa7-e90a-4819-8e78-ba123cd58cce","Type":"ContainerStarted","Data":"315a0f7a132b2ab04e7ac75fe697915b0a98ff6d3fdf50001fbcd4d4adaf5d2a"} Oct 30 19:24:19.523755 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:19.523734 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x79ns" event={"ID":"83041ead-f80a-4e66-9ef2-44b25467952f","Type":"ContainerStarted","Data":"ee9b88444d4f93ce2440c6b6ba3527ca9e6620a8bc1d508110b422462067d96f"} Oct 30 19:24:19.524911 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:19.524894 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dqztf" event={"ID":"a7d7990d-1d0f-41da-9dbf-1fbffe4af353","Type":"ContainerStarted","Data":"8abb9cdd72c205c969b92f61e96d8a8f8ff42513f80a6718bdb6ccf464ba5f6b"} Oct 30 19:24:19.525596 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:19.525566 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-driver\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:007ba215236827a40e1b33f3799d289065305ae886c05e51214d74d0ff24d62d\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a5e96bc702bc529b424e6c80f95d7d4fc4fe92f83df4a77c3b47eb3a30fe729\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"csi-liveness-probe\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c4db2cbbaecb77e6b3a4e8dd7ee7a4d4fa9ab69787e4330711a3abb1277af2ac\\\": ErrImagePull: pull QPS exceeded\"]" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqxgl" podUID="e1bb6034-1fd5-4a40-863d-e43314482f5c" Oct 30 19:24:19.526012 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:19.525992 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-672ql" event={"ID":"1165f4d9-846b-4f12-92c1-944193f28d21","Type":"ContainerStarted","Data":"0489dbb7ced385b338ddf87bce1d5b28a4ab00f111c79b47e1778a31b8d58b55"} Oct 30 19:24:19.527040 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:19.527019 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-147.us-west-2.compute.internal" event={"ID":"799ed54bc75d16c836f178a66c89fbdc","Type":"ContainerStarted","Data":"b54d0851972c77e8e6a51e74806c7af996506eef1df90a95cd4627860359c4a1"} Oct 30 19:24:19.528025 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:19.528002 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-l66t6" event={"ID":"c9eecef1-2e58-4444-8a51-25b0b18715f5","Type":"ContainerStarted","Data":"fb567e9b5fd55e8e41d6248cb814e780ec6e5c2490b72fadbbbdcaa8b2a8de4c"} Oct 30 19:24:19.529034 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:19.529012 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-p6vhl" event={"ID":"2198fe85-6a7d-4cb3-a9e1-f804851fdb1d","Type":"ContainerStarted","Data":"11f7ed69235aad4fde75385d30e20e7bcddeb953f8af831aaa10956067e145f3"} Oct 30 19:24:19.530139 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:19.530119 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" event={"ID":"cb4226c2-13b5-4ef8-97ba-ffd6126442f4","Type":"ContainerStarted","Data":"3cc73881ed59439fc78eb71b370e6d849851e175732e0f2187be3d1d42adaef6"} Oct 30 19:24:19.531182 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:19.531161 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rffx6" event={"ID":"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1","Type":"ContainerStarted","Data":"413c7b1cf9af624b23d5f9434258acc869486b4d718e815b2888be6ff87c381f"} Oct 30 19:24:19.532143 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:19.532124 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-147.us-west-2.compute.internal" event={"ID":"4cac457219e3215c4459c18e1cf7554f","Type":"ContainerStarted","Data":"f04dfc935e4256d3d2231f330ea198145045d2c6f31ece3289f9f72fbc680d4f"} Oct 30 19:24:19.708206 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:19.708137 2582 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:24:19.826362 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:19.826339 2582 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:24:19.838872 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:19.838816 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:19:18 +0000 UTC" deadline="2028-07-11 22:02:41.21033464 +0000 UTC" Oct 30 19:24:19.838872 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:19.838869 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14882h38m21.371468745s" Oct 30 19:24:20.304004 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:20.303784 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87-original-pull-secret\") pod \"global-pull-secret-syncer-kcr6x\" (UID: \"e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87\") " pod="kube-system/global-pull-secret-syncer-kcr6x" Oct 30 19:24:20.304004 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:20.303944 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:24:20.304004 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:20.304014 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87-original-pull-secret podName:e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87 nodeName:}" failed. No retries permitted until 2026-10-30 19:24:22.303991815 +0000 UTC m=+5.253171384 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87-original-pull-secret") pod "global-pull-secret-syncer-kcr6x" (UID: "e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:24:20.404711 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:20.404672 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adeacb06-6b16-4597-be99-ff84d8efdadc-metrics-certs\") pod \"network-metrics-daemon-mts99\" (UID: \"adeacb06-6b16-4597-be99-ff84d8efdadc\") " pod="openshift-multus/network-metrics-daemon-mts99" Oct 30 19:24:20.405222 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:20.404770 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj6h4\" (UniqueName: \"kubernetes.io/projected/6ea94667-080f-4d20-887c-376aee648406-kube-api-access-kj6h4\") pod \"network-check-target-t6m9p\" (UID: \"6ea94667-080f-4d20-887c-376aee648406\") " pod="openshift-network-diagnostics/network-check-target-t6m9p" Oct 30 19:24:20.405222 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:20.404933 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:24:20.405222 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:20.404959 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:24:20.405222 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:20.404975 2582 projected.go:196] Error preparing data for projected volume kube-api-access-kj6h4 for pod openshift-network-diagnostics/network-check-target-t6m9p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:24:20.405222 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:20.405032 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ea94667-080f-4d20-887c-376aee648406-kube-api-access-kj6h4 podName:6ea94667-080f-4d20-887c-376aee648406 nodeName:}" failed. No retries permitted until 2026-10-30 19:24:22.405014509 +0000 UTC m=+5.354194095 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-kj6h4" (UniqueName: "kubernetes.io/projected/6ea94667-080f-4d20-887c-376aee648406-kube-api-access-kj6h4") pod "network-check-target-t6m9p" (UID: "6ea94667-080f-4d20-887c-376aee648406") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:24:20.405222 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:20.405109 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:24:20.405222 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:20.405144 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/adeacb06-6b16-4597-be99-ff84d8efdadc-metrics-certs podName:adeacb06-6b16-4597-be99-ff84d8efdadc nodeName:}" failed. No retries permitted until 2026-10-30 19:24:22.405132636 +0000 UTC m=+5.354312211 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/adeacb06-6b16-4597-be99-ff84d8efdadc-metrics-certs") pod "network-metrics-daemon-mts99" (UID: "adeacb06-6b16-4597-be99-ff84d8efdadc") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:24:20.521329 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:20.521289 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mts99" Oct 30 19:24:20.521552 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:20.521523 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mts99" podUID="adeacb06-6b16-4597-be99-ff84d8efdadc" Oct 30 19:24:20.521654 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:20.521638 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t6m9p" Oct 30 19:24:20.521782 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:20.521761 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t6m9p" podUID="6ea94667-080f-4d20-887c-376aee648406" Oct 30 19:24:20.521931 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:20.521899 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kcr6x" Oct 30 19:24:20.522448 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:20.522419 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kcr6x" podUID="e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87" Oct 30 19:24:20.840045 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:20.839947 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:19:18 +0000 UTC" deadline="2028-05-22 19:47:41.415402358 +0000 UTC" Oct 30 19:24:20.840265 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:20.840049 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13680h23m20.575359233s" Oct 30 19:24:22.331980 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:22.331939 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87-original-pull-secret\") pod \"global-pull-secret-syncer-kcr6x\" (UID: \"e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87\") " pod="kube-system/global-pull-secret-syncer-kcr6x" Oct 30 19:24:22.332753 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:22.332166 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:24:22.332753 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:22.332223 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87-original-pull-secret podName:e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87 nodeName:}" failed. No retries permitted until 2026-10-30 19:24:26.332205996 +0000 UTC m=+9.281385580 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87-original-pull-secret") pod "global-pull-secret-syncer-kcr6x" (UID: "e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:24:22.433781 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:22.433261 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adeacb06-6b16-4597-be99-ff84d8efdadc-metrics-certs\") pod \"network-metrics-daemon-mts99\" (UID: \"adeacb06-6b16-4597-be99-ff84d8efdadc\") " pod="openshift-multus/network-metrics-daemon-mts99" Oct 30 19:24:22.433781 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:22.433346 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj6h4\" (UniqueName: \"kubernetes.io/projected/6ea94667-080f-4d20-887c-376aee648406-kube-api-access-kj6h4\") pod \"network-check-target-t6m9p\" (UID: \"6ea94667-080f-4d20-887c-376aee648406\") " pod="openshift-network-diagnostics/network-check-target-t6m9p" Oct 30 19:24:22.433781 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:22.433519 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:24:22.433781 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:22.433542 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:24:22.433781 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:22.433556 2582 projected.go:196] Error preparing data for projected volume kube-api-access-kj6h4 for pod openshift-network-diagnostics/network-check-target-t6m9p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:24:22.433781 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:22.433618 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ea94667-080f-4d20-887c-376aee648406-kube-api-access-kj6h4 podName:6ea94667-080f-4d20-887c-376aee648406 nodeName:}" failed. No retries permitted until 2026-10-30 19:24:26.433599085 +0000 UTC m=+9.382778666 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-kj6h4" (UniqueName: "kubernetes.io/projected/6ea94667-080f-4d20-887c-376aee648406-kube-api-access-kj6h4") pod "network-check-target-t6m9p" (UID: "6ea94667-080f-4d20-887c-376aee648406") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:24:22.433781 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:22.433700 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:24:22.433781 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:22.433746 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/adeacb06-6b16-4597-be99-ff84d8efdadc-metrics-certs podName:adeacb06-6b16-4597-be99-ff84d8efdadc nodeName:}" failed. No retries permitted until 2026-10-30 19:24:26.433732521 +0000 UTC m=+9.382912110 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/adeacb06-6b16-4597-be99-ff84d8efdadc-metrics-certs") pod "network-metrics-daemon-mts99" (UID: "adeacb06-6b16-4597-be99-ff84d8efdadc") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:24:22.517031 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:22.516997 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kcr6x" Oct 30 19:24:22.517216 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:22.517128 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kcr6x" podUID="e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87" Oct 30 19:24:22.517216 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:22.517201 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mts99" Oct 30 19:24:22.517372 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:22.517356 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t6m9p" Oct 30 19:24:22.517498 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:22.517448 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t6m9p" podUID="6ea94667-080f-4d20-887c-376aee648406" Oct 30 19:24:22.517563 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:22.517288 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mts99" podUID="adeacb06-6b16-4597-be99-ff84d8efdadc" Oct 30 19:24:23.582598 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:23.582510 2582 generic.go:356] "Generic (PLEG): container finished" podID="83041ead-f80a-4e66-9ef2-44b25467952f" containerID="48d796b1c1dd6ca68c018630e897ac3f363839b9b60b9e095a4773cc3da0f9f6" exitCode=0 Oct 30 19:24:23.582598 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:23.582581 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x79ns" event={"ID":"83041ead-f80a-4e66-9ef2-44b25467952f","Type":"ContainerDied","Data":"48d796b1c1dd6ca68c018630e897ac3f363839b9b60b9e095a4773cc3da0f9f6"} Oct 30 19:24:24.517649 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:24.517323 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kcr6x" Oct 30 19:24:24.517868 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:24.517728 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kcr6x" podUID="e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87" Oct 30 19:24:24.517868 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:24.517829 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mts99" Oct 30 19:24:24.517987 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:24.517918 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mts99" podUID="adeacb06-6b16-4597-be99-ff84d8efdadc" Oct 30 19:24:24.517987 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:24.517972 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t6m9p" Oct 30 19:24:24.518087 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:24.518042 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t6m9p" podUID="6ea94667-080f-4d20-887c-376aee648406" Oct 30 19:24:24.604417 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:24.603190 2582 generic.go:356] "Generic (PLEG): container finished" podID="4cac457219e3215c4459c18e1cf7554f" containerID="9e88e0a62e2eaf21a1a01be885c2225bc9d2403f5882e512e47279d5372e8b37" exitCode=0 Oct 30 19:24:24.604417 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:24.603299 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-147.us-west-2.compute.internal" event={"ID":"4cac457219e3215c4459c18e1cf7554f","Type":"ContainerDied","Data":"9e88e0a62e2eaf21a1a01be885c2225bc9d2403f5882e512e47279d5372e8b37"} Oct 30 19:24:24.611509 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:24.611478 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-147.us-west-2.compute.internal" event={"ID":"799ed54bc75d16c836f178a66c89fbdc","Type":"ContainerStarted","Data":"d85ea22b22cdc26966be2f5e80fcb7ed7a267b9672e6aeb56764db282fbe8138"} Oct 30 19:24:24.616531 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:24.615982 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-l66t6" event={"ID":"c9eecef1-2e58-4444-8a51-25b0b18715f5","Type":"ContainerStarted","Data":"1f097ad6e3fe5a502b6ea6286750e39161d7478ea8a8719812a9c4bbc62257d6"} Oct 30 19:24:24.858041 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:24.857982 2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-l66t6" Oct 30 19:24:24.859157 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:24.859132 2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-l66t6" Oct 30 19:24:24.969303 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:24.968911 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-147.us-west-2.compute.internal" podStartSLOduration=6.968894689 podStartE2EDuration="6.968894689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:24:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:24.849608621 +0000 UTC m=+7.798788213" watchObservedRunningTime="2026-10-30 19:24:24.968894689 +0000 UTC m=+7.918074280" Oct 30 19:24:25.088489 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:25.088391 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-l66t6" podStartSLOduration=2.109480943 podStartE2EDuration="7.088371712s" totalImagesPullingTime="4.978890769s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:24.969160801 +0000 UTC m=+7.918340393" watchObservedRunningTime="2026-10-30 19:24:25.088371712 +0000 UTC m=+8.037551302" Oct 30 19:24:25.621147 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:25.621111 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dqztf" event={"ID":"a7d7990d-1d0f-41da-9dbf-1fbffe4af353","Type":"ContainerStarted","Data":"74fe42b9838956c7d17a2bf51dc513e3c4fb6e7c91bb074f80e1f9c6efab586a"} Oct 30 19:24:25.625558 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:25.625525 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-147.us-west-2.compute.internal" event={"ID":"4cac457219e3215c4459c18e1cf7554f","Type":"ContainerStarted","Data":"4daf739aa186f41db284b47c40526b4053dc21abda6f82c250ac344046c54ab4"} Oct 30 19:24:25.891382 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:25.891243 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-dqztf" podStartSLOduration=1.80440857 podStartE2EDuration="7.891225104s" totalImagesPullingTime="6.086816534s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:24:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:25.749182063 +0000 UTC m=+8.698361661" watchObservedRunningTime="2026-10-30 19:24:25.891225104 +0000 UTC m=+8.840404698" Oct 30 19:24:25.892298 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:25.891970 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-147.us-west-2.compute.internal" podStartSLOduration=7.891958273 podStartE2EDuration="7.891958273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:24:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:25.890962997 +0000 UTC m=+8.840142588" watchObservedRunningTime="2026-10-30 19:24:25.891958273 +0000 UTC m=+8.841137864" Oct 30 19:24:26.364652 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:26.364597 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87-original-pull-secret\") pod \"global-pull-secret-syncer-kcr6x\" (UID: \"e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87\") " pod="kube-system/global-pull-secret-syncer-kcr6x" Oct 30 19:24:26.364841 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:26.364767 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:24:26.364841 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:26.364838 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87-original-pull-secret podName:e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87 nodeName:}" failed. No retries permitted until 2026-10-30 19:24:34.364819223 +0000 UTC m=+17.313998797 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87-original-pull-secret") pod "global-pull-secret-syncer-kcr6x" (UID: "e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:24:26.465486 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:26.465404 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj6h4\" (UniqueName: \"kubernetes.io/projected/6ea94667-080f-4d20-887c-376aee648406-kube-api-access-kj6h4\") pod \"network-check-target-t6m9p\" (UID: \"6ea94667-080f-4d20-887c-376aee648406\") " pod="openshift-network-diagnostics/network-check-target-t6m9p" Oct 30 19:24:26.465486 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:26.465476 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adeacb06-6b16-4597-be99-ff84d8efdadc-metrics-certs\") pod \"network-metrics-daemon-mts99\" (UID: \"adeacb06-6b16-4597-be99-ff84d8efdadc\") " pod="openshift-multus/network-metrics-daemon-mts99" Oct 30 19:24:26.465703 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:26.465589 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:24:26.465703 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:26.465645 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/adeacb06-6b16-4597-be99-ff84d8efdadc-metrics-certs podName:adeacb06-6b16-4597-be99-ff84d8efdadc nodeName:}" failed. No retries permitted until 2026-10-30 19:24:34.46562879 +0000 UTC m=+17.414808363 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/adeacb06-6b16-4597-be99-ff84d8efdadc-metrics-certs") pod "network-metrics-daemon-mts99" (UID: "adeacb06-6b16-4597-be99-ff84d8efdadc") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:24:26.465817 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:26.465726 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:24:26.465817 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:26.465744 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:24:26.465817 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:26.465755 2582 projected.go:196] Error preparing data for projected volume kube-api-access-kj6h4 for pod openshift-network-diagnostics/network-check-target-t6m9p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:24:26.465817 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:26.465789 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ea94667-080f-4d20-887c-376aee648406-kube-api-access-kj6h4 podName:6ea94667-080f-4d20-887c-376aee648406 nodeName:}" failed. No retries permitted until 2026-10-30 19:24:34.465777447 +0000 UTC m=+17.414957015 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-kj6h4" (UniqueName: "kubernetes.io/projected/6ea94667-080f-4d20-887c-376aee648406-kube-api-access-kj6h4") pod "network-check-target-t6m9p" (UID: "6ea94667-080f-4d20-887c-376aee648406") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:24:26.517731 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:26.517599 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mts99" Oct 30 19:24:26.517890 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:26.517737 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mts99" podUID="adeacb06-6b16-4597-be99-ff84d8efdadc" Oct 30 19:24:26.517890 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:26.517808 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t6m9p" Oct 30 19:24:26.517890 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:26.517858 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t6m9p" podUID="6ea94667-080f-4d20-887c-376aee648406" Oct 30 19:24:26.517890 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:26.517886 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kcr6x" Oct 30 19:24:26.518042 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:26.517927 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kcr6x" podUID="e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87" Oct 30 19:24:26.627832 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:26.627070 2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:24:27.632999 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:27.632284 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rffx6" event={"ID":"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1","Type":"ContainerStarted","Data":"2a4a0d179f00152a21d1439464b8f83abfd11257a85b66467fbb9ce4b4310191"} Oct 30 19:24:27.755093 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:27.754997 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-rffx6" podStartSLOduration=1.514111374 podStartE2EDuration="9.754981499s" totalImagesPullingTime="8.240870125s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:24:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:27.754805006 +0000 UTC m=+10.703984664" watchObservedRunningTime="2026-10-30 19:24:27.754981499 +0000 UTC m=+10.704161091" Oct 30 19:24:28.516996 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:28.516925 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kcr6x" Oct 30 19:24:28.517157 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:28.516925 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t6m9p" Oct 30 19:24:28.517157 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:28.517045 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kcr6x" podUID="e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87" Oct 30 19:24:28.517157 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:28.517054 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mts99" Oct 30 19:24:28.517157 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:28.517141 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t6m9p" podUID="6ea94667-080f-4d20-887c-376aee648406" Oct 30 19:24:28.517375 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:28.517203 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mts99" podUID="adeacb06-6b16-4597-be99-ff84d8efdadc" Oct 30 19:24:28.636020 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:28.635978 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-p6vhl" event={"ID":"2198fe85-6a7d-4cb3-a9e1-f804851fdb1d","Type":"ContainerStarted","Data":"25bea625d610395429c8577552e682b73297c8a64596158bb5d50d5bf076e3bf"} Oct 30 19:24:28.638308 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:28.638283 2582 generic.go:356] "Generic (PLEG): container finished" podID="83041ead-f80a-4e66-9ef2-44b25467952f" containerID="902ac762dd9530b7790f0c04fd948845b7fc991be2051f83050257f674dbb593" exitCode=0 Oct 30 19:24:28.638463 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:28.638389 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x79ns" event={"ID":"83041ead-f80a-4e66-9ef2-44b25467952f","Type":"ContainerDied","Data":"902ac762dd9530b7790f0c04fd948845b7fc991be2051f83050257f674dbb593"} Oct 30 19:24:28.755699 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:28.755637 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-p6vhl" podStartSLOduration=2.04074068 podStartE2EDuration="10.75561951s" totalImagesPullingTime="8.71487883s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:24:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:28.755023105 +0000 UTC m=+11.704202714" watchObservedRunningTime="2026-10-30 19:24:28.75561951 +0000 UTC m=+11.704799101" Oct 30 19:24:29.643670 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:29.643627 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-s5pd4" event={"ID":"fd873aa7-e90a-4819-8e78-ba123cd58cce","Type":"ContainerStarted","Data":"677af01f148045f9351bfd56d336f5df1653ad411bf36208067cc6b45468f3df"} Oct 30 19:24:29.646438 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:29.646409 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-672ql" event={"ID":"1165f4d9-846b-4f12-92c1-944193f28d21","Type":"ContainerStarted","Data":"525e77541daae470a82cc8154e37a0bf806a42c0101289c1e7994313139bfb1a"} Oct 30 19:24:29.762395 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:29.762282 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-s5pd4" podStartSLOduration=2.77643337 podStartE2EDuration="11.762267373s" totalImagesPullingTime="8.985834003s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:24:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:29.761832629 +0000 UTC m=+12.711012220" watchObservedRunningTime="2026-10-30 19:24:29.762267373 +0000 UTC m=+12.711446961" Oct 30 19:24:29.886310 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:29.886258 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-672ql" podStartSLOduration=1.447123031 podStartE2EDuration="11.886239287s" totalImagesPullingTime="10.439116256s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:24:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:29.885924204 +0000 UTC m=+12.835103798" watchObservedRunningTime="2026-10-30 19:24:29.886239287 +0000 UTC m=+12.835418878" Oct 30 19:24:30.517154 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:30.517115 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kcr6x" Oct 30 19:24:30.517366 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:30.517196 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t6m9p" Oct 30 19:24:30.517366 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:30.517288 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mts99" Oct 30 19:24:30.517366 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:30.517310 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kcr6x" podUID="e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87" Oct 30 19:24:30.517532 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:30.517389 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mts99" podUID="adeacb06-6b16-4597-be99-ff84d8efdadc" Oct 30 19:24:30.517532 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:30.517500 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t6m9p" podUID="6ea94667-080f-4d20-887c-376aee648406" Oct 30 19:24:30.650059 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:30.650019 2582 generic.go:356] "Generic (PLEG): container finished" podID="83041ead-f80a-4e66-9ef2-44b25467952f" containerID="4fd5532f313fe8a9297265f6a23b0766134965eba498f8045bf5a8f4d160faad" exitCode=0 Oct 30 19:24:30.650551 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:30.650064 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x79ns" event={"ID":"83041ead-f80a-4e66-9ef2-44b25467952f","Type":"ContainerDied","Data":"4fd5532f313fe8a9297265f6a23b0766134965eba498f8045bf5a8f4d160faad"} Oct 30 19:24:32.516899 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:32.516705 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mts99" Oct 30 19:24:32.517491 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:32.516705 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t6m9p" Oct 30 19:24:32.517491 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:32.517004 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mts99" podUID="adeacb06-6b16-4597-be99-ff84d8efdadc" Oct 30 19:24:32.517491 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:32.516705 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kcr6x" Oct 30 19:24:32.517491 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:32.517080 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t6m9p" podUID="6ea94667-080f-4d20-887c-376aee648406" Oct 30 19:24:32.517491 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:32.517160 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kcr6x" podUID="e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87" Oct 30 19:24:32.655931 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:32.655896 2582 generic.go:356] "Generic (PLEG): container finished" podID="83041ead-f80a-4e66-9ef2-44b25467952f" containerID="ab599efcac7bc5bd357389053c803c6bebfd00caebf70d46d4cd830e846ae144" exitCode=0 Oct 30 19:24:32.656077 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:32.655931 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x79ns" event={"ID":"83041ead-f80a-4e66-9ef2-44b25467952f","Type":"ContainerDied","Data":"ab599efcac7bc5bd357389053c803c6bebfd00caebf70d46d4cd830e846ae144"} Oct 30 19:24:34.430409 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:34.430214 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87-original-pull-secret\") pod \"global-pull-secret-syncer-kcr6x\" (UID: \"e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87\") " pod="kube-system/global-pull-secret-syncer-kcr6x" Oct 30 19:24:34.431069 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:34.430433 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:24:34.431069 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:34.430493 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87-original-pull-secret podName:e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87 nodeName:}" failed. No retries permitted until 2026-10-30 19:24:50.430476629 +0000 UTC m=+33.379656211 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87-original-pull-secret") pod "global-pull-secret-syncer-kcr6x" (UID: "e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:24:34.517176 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:34.517153 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mts99" Oct 30 19:24:34.517301 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:34.517254 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t6m9p" Oct 30 19:24:34.517301 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:34.517265 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kcr6x" Oct 30 19:24:34.517301 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:34.517268 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mts99" podUID="adeacb06-6b16-4597-be99-ff84d8efdadc" Oct 30 19:24:34.517532 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:34.517339 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t6m9p" podUID="6ea94667-080f-4d20-887c-376aee648406" Oct 30 19:24:34.517532 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:34.517438 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kcr6x" podUID="e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87" Oct 30 19:24:34.531272 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:34.531251 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj6h4\" (UniqueName: \"kubernetes.io/projected/6ea94667-080f-4d20-887c-376aee648406-kube-api-access-kj6h4\") pod \"network-check-target-t6m9p\" (UID: \"6ea94667-080f-4d20-887c-376aee648406\") " pod="openshift-network-diagnostics/network-check-target-t6m9p" Oct 30 19:24:34.531402 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:34.531293 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adeacb06-6b16-4597-be99-ff84d8efdadc-metrics-certs\") pod \"network-metrics-daemon-mts99\" (UID: \"adeacb06-6b16-4597-be99-ff84d8efdadc\") " pod="openshift-multus/network-metrics-daemon-mts99" Oct 30 19:24:34.531471 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:34.531399 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:24:34.531471 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:34.531424 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:24:34.531471 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:34.531442 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:24:34.531471 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:34.531449 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/adeacb06-6b16-4597-be99-ff84d8efdadc-metrics-certs podName:adeacb06-6b16-4597-be99-ff84d8efdadc nodeName:}" failed. No retries permitted until 2026-10-30 19:24:50.531434084 +0000 UTC m=+33.480613657 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/adeacb06-6b16-4597-be99-ff84d8efdadc-metrics-certs") pod "network-metrics-daemon-mts99" (UID: "adeacb06-6b16-4597-be99-ff84d8efdadc") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:24:34.531471 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:34.531452 2582 projected.go:196] Error preparing data for projected volume kube-api-access-kj6h4 for pod openshift-network-diagnostics/network-check-target-t6m9p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:24:34.531726 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:34.531487 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ea94667-080f-4d20-887c-376aee648406-kube-api-access-kj6h4 podName:6ea94667-080f-4d20-887c-376aee648406 nodeName:}" failed. No retries permitted until 2026-10-30 19:24:50.531475113 +0000 UTC m=+33.480654682 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-kj6h4" (UniqueName: "kubernetes.io/projected/6ea94667-080f-4d20-887c-376aee648406-kube-api-access-kj6h4") pod "network-check-target-t6m9p" (UID: "6ea94667-080f-4d20-887c-376aee648406") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:24:34.665521 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:34.665487 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" event={"ID":"cb4226c2-13b5-4ef8-97ba-ffd6126442f4","Type":"ContainerStarted","Data":"f8e9233d93746fc75591c4afeadde64b7cd020734a7ed30df4e0b56d27347791"} Oct 30 19:24:34.665649 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:34.665594 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" event={"ID":"cb4226c2-13b5-4ef8-97ba-ffd6126442f4","Type":"ContainerStarted","Data":"0b703db1e64efbeb8e49c8746b95d8fc3f7fe9e3e110f853d604a52d4f3d3c36"} Oct 30 19:24:34.665649 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:34.665644 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" event={"ID":"cb4226c2-13b5-4ef8-97ba-ffd6126442f4","Type":"ContainerStarted","Data":"f83b9e17acc424722b0ebe14afc0b28b713683c71648c358f39cdfe82bce05b2"} Oct 30 19:24:34.665760 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:34.665690 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" event={"ID":"cb4226c2-13b5-4ef8-97ba-ffd6126442f4","Type":"ContainerStarted","Data":"e80aa320d83356384e60ae9f0190c74ce9a72b0bd12d417e0bfa06ca035d7a8e"} Oct 30 19:24:34.665760 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:34.665737 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" event={"ID":"cb4226c2-13b5-4ef8-97ba-ffd6126442f4","Type":"ContainerStarted","Data":"4fcb8089b5a35fafcdca59411dfe4c6430c536d83a4950561f19d1a03fbe5ed9"} Oct 30 19:24:34.665865 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:34.665781 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" event={"ID":"cb4226c2-13b5-4ef8-97ba-ffd6126442f4","Type":"ContainerStarted","Data":"de0d5b35bdd974d049ffce8b9436b96b6a08da728128019de60b0096dcd39f8e"} Oct 30 19:24:35.670696 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:35.670658 2582 generic.go:356] "Generic (PLEG): container finished" podID="83041ead-f80a-4e66-9ef2-44b25467952f" containerID="eb6a30a0d3ce419d6c8ebe8a9d3851f8933f97adc15d76adc584e9181d5dd4b7" exitCode=0 Oct 30 19:24:35.671227 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:35.670706 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x79ns" event={"ID":"83041ead-f80a-4e66-9ef2-44b25467952f","Type":"ContainerDied","Data":"eb6a30a0d3ce419d6c8ebe8a9d3851f8933f97adc15d76adc584e9181d5dd4b7"} Oct 30 19:24:36.517077 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:36.517040 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t6m9p" Oct 30 19:24:36.517077 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:36.517064 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kcr6x" Oct 30 19:24:36.517305 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:36.517171 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t6m9p" podUID="6ea94667-080f-4d20-887c-376aee648406" Oct 30 19:24:36.517305 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:36.517208 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mts99" Oct 30 19:24:36.517305 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:36.517294 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kcr6x" podUID="e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87" Oct 30 19:24:36.517436 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:36.517375 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mts99" podUID="adeacb06-6b16-4597-be99-ff84d8efdadc" Oct 30 19:24:36.677768 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:36.677738 2582 generic.go:356] "Generic (PLEG): container finished" podID="83041ead-f80a-4e66-9ef2-44b25467952f" containerID="9a0a76baae4cb66e83e2cfc02fa75d29984cc381575f3f000f0b3c521e6180a6" exitCode=0 Oct 30 19:24:36.678093 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:36.677810 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x79ns" event={"ID":"83041ead-f80a-4e66-9ef2-44b25467952f","Type":"ContainerDied","Data":"9a0a76baae4cb66e83e2cfc02fa75d29984cc381575f3f000f0b3c521e6180a6"} Oct 30 19:24:37.683226 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:37.683039 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" event={"ID":"cb4226c2-13b5-4ef8-97ba-ffd6126442f4","Type":"ContainerStarted","Data":"f24797b9404b7f60d8bd3402099415c91156b7d4c278e8bc3fee06894d53d46c"} Oct 30 19:24:37.684480 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:37.684451 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqxgl" event={"ID":"e1bb6034-1fd5-4a40-863d-e43314482f5c","Type":"ContainerStarted","Data":"aaa87d6a2efd494e6797007fb8d5339c6314a6e0eaf050d2e6783f09527a7b34"} Oct 30 19:24:37.687654 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:37.687631 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x79ns" event={"ID":"83041ead-f80a-4e66-9ef2-44b25467952f","Type":"ContainerStarted","Data":"8dd977db516aa588d63811b3c04311172eff812ca7139a82d0ec12b36db2603c"} Oct 30 19:24:37.814117 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:37.814069 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-x79ns" podStartSLOduration=7.520560088 podStartE2EDuration="19.814056437s" totalImagesPullingTime="12.293496349s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:24:18 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:37.813925266 +0000 UTC m=+20.763104856" watchObservedRunningTime="2026-10-30 19:24:37.814056437 +0000 UTC m=+20.763236027" Oct 30 19:24:38.210503 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:38.210451 2582 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 19:24:38.467064 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:38.466887 2582 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T19:24:38.210468384Z","UUID":"251942dd-62b7-4f64-85a4-99dee93fdc84","Handler":null,"Name":"","Endpoint":""} Oct 30 19:24:38.468094 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:38.468074 2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 19:24:38.468234 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:38.468100 2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 19:24:38.517194 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:38.517164 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mts99" Oct 30 19:24:38.517344 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:38.517195 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kcr6x" Oct 30 19:24:38.517344 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:38.517287 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mts99" podUID="adeacb06-6b16-4597-be99-ff84d8efdadc" Oct 30 19:24:38.517344 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:38.517329 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t6m9p" Oct 30 19:24:38.517497 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:38.517417 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t6m9p" podUID="6ea94667-080f-4d20-887c-376aee648406" Oct 30 19:24:38.517497 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:38.517483 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kcr6x" podUID="e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87" Oct 30 19:24:38.693858 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:38.693824 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqxgl" event={"ID":"e1bb6034-1fd5-4a40-863d-e43314482f5c","Type":"ContainerStarted","Data":"d3bc4bc1dd788451a6c53610cb575d5796d06fc9d6d672038f810553f5e1cca6"} Oct 30 19:24:39.699649 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:39.699614 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" event={"ID":"cb4226c2-13b5-4ef8-97ba-ffd6126442f4","Type":"ContainerStarted","Data":"3217cff66f97d7750b79f819101b9261cfcb6cf0571f9d66510fc56222386ad2"} Oct 30 19:24:39.700316 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:39.699822 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:39.702131 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:39.702109 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqxgl" event={"ID":"e1bb6034-1fd5-4a40-863d-e43314482f5c","Type":"ContainerStarted","Data":"4f6a5d8ccf5f7908c728f25187d9e07df09e919f7f89551d8a7b91963a7def8c"} Oct 30 19:24:39.714532 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:39.714508 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:39.990138 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:39.989720 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" podStartSLOduration=6.876813946 podStartE2EDuration="21.989701407s" totalImagesPullingTime="15.112887461s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:24:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:39.989182042 +0000 UTC m=+22.938361634" watchObservedRunningTime="2026-10-30 19:24:39.989701407 +0000 UTC m=+22.938881000" Oct 30 19:24:40.248795 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:40.248701 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqxgl" podStartSLOduration=1.9066209029999999 podStartE2EDuration="22.248687396s" totalImagesPullingTime="20.342066493s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:24:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=3 observedRunningTime="2026-10-30 19:24:40.248163034 +0000 UTC m=+23.197342624" watchObservedRunningTime="2026-10-30 19:24:40.248687396 +0000 UTC m=+23.197867012" Oct 30 19:24:40.517605 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:40.517530 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t6m9p" Oct 30 19:24:40.517605 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:40.517564 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mts99" Oct 30 19:24:40.517825 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:40.517621 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t6m9p" podUID="6ea94667-080f-4d20-887c-376aee648406" Oct 30 19:24:40.517825 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:40.517664 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kcr6x" Oct 30 19:24:40.517825 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:40.517774 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mts99" podUID="adeacb06-6b16-4597-be99-ff84d8efdadc" Oct 30 19:24:40.517960 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:40.517880 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kcr6x" podUID="e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87" Oct 30 19:24:40.704911 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:40.704875 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:40.705296 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:40.704943 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:40.718933 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:40.718910 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:40.735216 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:40.735185 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-l66t6" Oct 30 19:24:40.735387 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:40.735312 2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:24:40.735972 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:40.735945 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-l66t6" Oct 30 19:24:41.372830 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:41.372609 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kcr6x"] Oct 30 19:24:41.373079 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:41.372925 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kcr6x" Oct 30 19:24:41.373079 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:41.373032 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kcr6x" podUID="e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87" Oct 30 19:24:41.375647 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:41.375622 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-t6m9p"] Oct 30 19:24:41.375760 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:41.375716 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t6m9p" Oct 30 19:24:41.375814 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:41.375799 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t6m9p" podUID="6ea94667-080f-4d20-887c-376aee648406" Oct 30 19:24:41.376103 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:41.376086 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mts99"] Oct 30 19:24:41.376181 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:41.376171 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mts99" Oct 30 19:24:41.376265 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:41.376251 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mts99" podUID="adeacb06-6b16-4597-be99-ff84d8efdadc" Oct 30 19:24:43.517326 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:43.517296 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kcr6x" Oct 30 19:24:43.517326 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:43.517326 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mts99" Oct 30 19:24:43.517739 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:43.517296 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t6m9p" Oct 30 19:24:43.517739 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:43.517444 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kcr6x" podUID="e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87" Oct 30 19:24:43.517739 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:43.517515 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mts99" podUID="adeacb06-6b16-4597-be99-ff84d8efdadc" Oct 30 19:24:43.517739 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:43.517613 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t6m9p" podUID="6ea94667-080f-4d20-887c-376aee648406" Oct 30 19:24:45.517584 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:45.517535 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mts99" Oct 30 19:24:45.517584 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:45.517535 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t6m9p" Oct 30 19:24:45.517981 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:45.517748 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mts99" podUID="adeacb06-6b16-4597-be99-ff84d8efdadc" Oct 30 19:24:45.517981 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:45.517774 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kcr6x" Oct 30 19:24:45.517981 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:45.517851 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t6m9p" podUID="6ea94667-080f-4d20-887c-376aee648406" Oct 30 19:24:45.517981 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:45.517918 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kcr6x" podUID="e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87" Oct 30 19:24:47.376924 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:47.376899 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-147.us-west-2.compute.internal" event="NodeReady" Oct 30 19:24:47.377378 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:47.377018 2582 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 19:24:47.517845 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:47.517816 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t6m9p" Oct 30 19:24:47.518014 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:47.517981 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kcr6x" Oct 30 19:24:47.518168 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:47.518147 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mts99" Oct 30 19:24:47.573782 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:47.573732 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 19:24:47.573782 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:47.573760 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 19:24:47.573928 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:47.573841 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-vwkbz\"" Oct 30 19:24:47.573928 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:47.573845 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9jbqb\"" Oct 30 19:24:47.573928 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:47.573862 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 19:24:47.573928 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:47.573893 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4g88n\"" Oct 30 19:24:47.624931 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:47.624904 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 19:24:49.196366 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:49.196335 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-p6vhl_2198fe85-6a7d-4cb3-a9e1-f804851fdb1d/dns-node-resolver/0.log" Oct 30 19:24:49.815329 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:49.815303 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dqztf_a7d7990d-1d0f-41da-9dbf-1fbffe4af353/node-ca/0.log" Oct 30 19:24:50.448624 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:50.448589 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87-original-pull-secret\") pod \"global-pull-secret-syncer-kcr6x\" (UID: \"e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87\") " pod="kube-system/global-pull-secret-syncer-kcr6x" Oct 30 19:24:50.451892 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:50.451871 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87-original-pull-secret\") pod \"global-pull-secret-syncer-kcr6x\" (UID: \"e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87\") " pod="kube-system/global-pull-secret-syncer-kcr6x" Oct 30 19:24:50.535248 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:50.535214 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kcr6x" Oct 30 19:24:50.549603 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:50.549576 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj6h4\" (UniqueName: \"kubernetes.io/projected/6ea94667-080f-4d20-887c-376aee648406-kube-api-access-kj6h4\") pod \"network-check-target-t6m9p\" (UID: \"6ea94667-080f-4d20-887c-376aee648406\") " pod="openshift-network-diagnostics/network-check-target-t6m9p" Oct 30 19:24:50.549713 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:50.549626 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adeacb06-6b16-4597-be99-ff84d8efdadc-metrics-certs\") pod \"network-metrics-daemon-mts99\" (UID: \"adeacb06-6b16-4597-be99-ff84d8efdadc\") " pod="openshift-multus/network-metrics-daemon-mts99" Oct 30 19:24:50.549776 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:50.549715 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 19:24:50.549828 ip-10-0-131-147 kubenswrapper[2582]: E1030 19:24:50.549786 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/adeacb06-6b16-4597-be99-ff84d8efdadc-metrics-certs podName:adeacb06-6b16-4597-be99-ff84d8efdadc nodeName:}" failed. No retries permitted until 2026-10-30 19:25:22.549758421 +0000 UTC m=+65.498938011 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/adeacb06-6b16-4597-be99-ff84d8efdadc-metrics-certs") pod "network-metrics-daemon-mts99" (UID: "adeacb06-6b16-4597-be99-ff84d8efdadc") : secret "metrics-daemon-secret" not found Oct 30 19:24:50.552184 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:50.552164 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj6h4\" (UniqueName: \"kubernetes.io/projected/6ea94667-080f-4d20-887c-376aee648406-kube-api-access-kj6h4\") pod \"network-check-target-t6m9p\" (UID: \"6ea94667-080f-4d20-887c-376aee648406\") " pod="openshift-network-diagnostics/network-check-target-t6m9p" Oct 30 19:24:50.830103 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:50.830076 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t6m9p" Oct 30 19:24:50.919618 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:50.919550 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kcr6x"] Oct 30 19:24:50.922332 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:50.922297 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6aabcd8_9a8c_4e92_9f6f_5b001cea8e87.slice/crio-659500fc8ef82e9b0ecf07fbac2742705e56a3583a37691572f5d52dbd2293fa WatchSource:0}: Error finding container 659500fc8ef82e9b0ecf07fbac2742705e56a3583a37691572f5d52dbd2293fa: Status 404 returned error can't find the container with id 659500fc8ef82e9b0ecf07fbac2742705e56a3583a37691572f5d52dbd2293fa Oct 30 19:24:51.252826 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:51.252753 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-t6m9p"] Oct 30 19:24:51.255715 ip-10-0-131-147 kubenswrapper[2582]: W1030 19:24:51.255691 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ea94667_080f_4d20_887c_376aee648406.slice/crio-25b3880037d92a8f60e1c2d3ff001dd298cfefe4eab2e51530d71980d13f6825 WatchSource:0}: Error finding container 25b3880037d92a8f60e1c2d3ff001dd298cfefe4eab2e51530d71980d13f6825: Status 404 returned error can't find the container with id 25b3880037d92a8f60e1c2d3ff001dd298cfefe4eab2e51530d71980d13f6825 Oct 30 19:24:51.728424 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:51.728381 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-t6m9p" event={"ID":"6ea94667-080f-4d20-887c-376aee648406","Type":"ContainerStarted","Data":"25b3880037d92a8f60e1c2d3ff001dd298cfefe4eab2e51530d71980d13f6825"} Oct 30 19:24:51.729458 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:51.729427 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kcr6x" event={"ID":"e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87","Type":"ContainerStarted","Data":"659500fc8ef82e9b0ecf07fbac2742705e56a3583a37691572f5d52dbd2293fa"} Oct 30 19:24:55.740931 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:55.740853 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-t6m9p" event={"ID":"6ea94667-080f-4d20-887c-376aee648406","Type":"ContainerStarted","Data":"44e878a1ca461a10556d827f1890f723f6b68ed5537107cb34e0490c09e484c1"} Oct 30 19:24:55.741412 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:55.740988 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-t6m9p" Oct 30 19:24:55.859324 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:55.859277 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-t6m9p" podStartSLOduration=33.833430409 podStartE2EDuration="37.859265442s" totalImagesPullingTime="4.025835033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:55.859200373 +0000 UTC m=+38.808379961" watchObservedRunningTime="2026-10-30 19:24:55.859265442 +0000 UTC m=+38.808445030" Oct 30 19:24:56.411789 ip-10-0-131-147 kubenswrapper[2582]: I1030 19:24:56.411691 2582 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 19:24:56.411831 ip-10-0-131-147 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 19:24:56.420812 ip-10-0-131-147 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 19:24:56.420961 ip-10-0-131-147 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 19:24:56.421006 ip-10-0-131-147 systemd[1]: kubelet.service: Consumed 2.083s CPU time. Oct 30 19:24:56.431821 ip-10-0-131-147 systemd[1]: Starting Kubernetes Kubelet... Oct 30 19:24:56.510566 ip-10-0-131-147 kubenswrapper[4292]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:24:56.510566 ip-10-0-131-147 kubenswrapper[4292]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 19:24:56.510566 ip-10-0-131-147 kubenswrapper[4292]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:24:56.510566 ip-10-0-131-147 kubenswrapper[4292]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:24:56.512597 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512523 4292 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:24:56.512597 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512596 4292 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:24:56.512597 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512600 4292 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:24:56.512692 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512603 4292 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:24:56.512692 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512606 4292 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:24:56.512692 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512609 4292 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:24:56.512692 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512611 4292 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:24:56.512692 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512614 4292 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:24:56.512692 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512617 4292 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:24:56.512692 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512619 4292 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:24:56.512692 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512622 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:24:56.512692 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512625 4292 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:24:56.512692 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512628 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:24:56.512692 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512631 4292 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:24:56.512692 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512633 4292 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:24:56.512692 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512636 4292 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:24:56.512692 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512639 4292 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:24:56.512692 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512642 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:24:56.512692 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512644 4292 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:24:56.512692 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512647 4292 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:24:56.512692 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512654 4292 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:24:56.512692 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512657 4292 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:24:56.512692 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512660 4292 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:24:56.513166 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512663 4292 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:24:56.513166 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512666 4292 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:24:56.513166 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512669 4292 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:24:56.513166 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512671 4292 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:24:56.513166 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512674 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:24:56.513166 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512677 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:24:56.513166 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512680 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:24:56.513166 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512682 4292 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:24:56.513166 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512685 4292 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:24:56.513166 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512688 4292 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:24:56.513166 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512690 4292 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:24:56.513166 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512693 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:24:56.513166 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512696 4292 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:24:56.513166 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512698 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:24:56.513166 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512701 4292 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:24:56.513166 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512704 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:24:56.513166 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512707 4292 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:24:56.513166 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512709 4292 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:24:56.513166 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512712 4292 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:24:56.513166 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512714 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:24:56.513689 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512717 4292 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:24:56.513689 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512719 4292 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:24:56.513689 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512723 4292 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:24:56.513689 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512728 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:24:56.513689 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512731 4292 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:24:56.513689 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512734 4292 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:24:56.513689 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512737 4292 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:24:56.513689 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512739 4292 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:24:56.513689 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512742 4292 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:24:56.513689 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512745 4292 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:24:56.513689 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512748 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:24:56.513689 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512750 4292 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:24:56.513689 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512753 4292 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:24:56.513689 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512755 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:24:56.513689 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512758 4292 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:24:56.513689 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512760 4292 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:24:56.513689 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512763 4292 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:24:56.513689 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512767 4292 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:24:56.513689 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512769 4292 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:24:56.514152 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512772 4292 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:24:56.514152 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512775 4292 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:24:56.514152 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512777 4292 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:24:56.514152 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512780 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:24:56.514152 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512782 4292 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:24:56.514152 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512785 4292 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:24:56.514152 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512788 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:24:56.514152 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512790 4292 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:24:56.514152 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512793 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:24:56.514152 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512795 4292 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:24:56.514152 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512798 4292 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:24:56.514152 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512801 4292 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:24:56.514152 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512803 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:24:56.514152 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512805 4292 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:24:56.514152 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512808 4292 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:24:56.514152 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512810 4292 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:24:56.514152 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512813 4292 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:24:56.514152 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512815 4292 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:24:56.514152 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512818 4292 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:24:56.514152 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512820 4292 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:24:56.514665 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512823 4292 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:24:56.514665 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512826 4292 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:24:56.514665 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512829 4292 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:24:56.514665 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512832 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:24:56.514665 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512834 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:24:56.514665 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512837 4292 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:24:56.514665 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512839 4292 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:24:56.514665 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512841 4292 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:24:56.514665 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512844 4292 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:24:56.514665 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512846 4292 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:24:56.514665 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512849 4292 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:24:56.514665 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512852 4292 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:24:56.514665 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512855 4292 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:24:56.514665 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512857 4292 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:24:56.514665 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512860 4292 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:24:56.514665 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512862 4292 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:24:56.514665 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512865 4292 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:24:56.514665 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512867 4292 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:24:56.514665 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512870 4292 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:24:56.515120 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512874 4292 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:24:56.515120 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512877 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:24:56.515120 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512881 4292 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:24:56.515120 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512884 4292 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:24:56.515120 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.512888 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:24:56.515120 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513682 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:24:56.515120 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513690 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:24:56.515120 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513693 4292 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:24:56.515120 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513696 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:24:56.515120 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513699 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:24:56.515120 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513704 4292 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:24:56.515120 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513708 4292 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:24:56.515120 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513711 4292 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:24:56.515120 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513714 4292 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:24:56.515120 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513717 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:24:56.515120 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513720 4292 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:24:56.515120 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513723 4292 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:24:56.515120 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513725 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:24:56.515570 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513728 4292 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:24:56.515570 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513730 4292 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:24:56.515570 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513733 4292 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:24:56.515570 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513735 4292 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:24:56.515570 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513738 4292 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:24:56.515570 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513740 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:24:56.515570 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513744 4292 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:24:56.515570 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513747 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:24:56.515570 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513749 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:24:56.515570 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513752 4292 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:24:56.515570 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513754 4292 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:24:56.515570 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513757 4292 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:24:56.515570 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513760 4292 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:24:56.515570 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513763 4292 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:24:56.515570 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513766 4292 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:24:56.515570 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513768 4292 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:24:56.515570 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513771 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:24:56.515570 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513780 4292 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:24:56.515570 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513783 4292 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:24:56.515570 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513786 4292 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:24:56.516047 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513788 4292 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:24:56.516047 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513791 4292 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:24:56.516047 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513794 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:24:56.516047 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513796 4292 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:24:56.516047 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513799 4292 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:24:56.516047 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513802 4292 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:24:56.516047 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513805 4292 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:24:56.516047 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513807 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:24:56.516047 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513810 4292 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:24:56.516047 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513812 4292 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:24:56.516047 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513815 4292 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:24:56.516047 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513818 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:24:56.516047 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513820 4292 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:24:56.516047 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513823 4292 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:24:56.516047 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513825 4292 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:24:56.516047 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513827 4292 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:24:56.516047 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513830 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:24:56.516047 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513833 4292 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:24:56.516047 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513836 4292 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:24:56.516047 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513839 4292 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:24:56.516530 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513841 4292 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:24:56.516530 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513844 4292 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:24:56.516530 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513848 4292 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:24:56.516530 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513852 4292 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:24:56.516530 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513855 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:24:56.516530 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513858 4292 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:24:56.516530 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513860 4292 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:24:56.516530 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513863 4292 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:24:56.516530 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513865 4292 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:24:56.516530 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513868 4292 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:24:56.516530 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513871 4292 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:24:56.516530 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513874 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:24:56.516530 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513876 4292 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:24:56.516530 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513879 4292 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:24:56.516530 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513882 4292 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:24:56.516530 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513884 4292 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:24:56.516530 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513887 4292 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:24:56.516530 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513889 4292 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:24:56.516530 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513892 4292 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:24:56.517025 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513894 4292 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:24:56.517025 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513897 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:24:56.517025 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513899 4292 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:24:56.517025 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513902 4292 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:24:56.517025 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513905 4292 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:24:56.517025 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513907 4292 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:24:56.517025 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513910 4292 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:24:56.517025 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513913 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:24:56.517025 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513916 4292 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:24:56.517025 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513918 4292 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:24:56.517025 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513921 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:24:56.517025 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513923 4292 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:24:56.517025 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513926 4292 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:24:56.517025 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513929 4292 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:24:56.517025 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513931 4292 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:24:56.517025 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513934 4292 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:24:56.517025 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513936 4292 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:24:56.517025 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513939 4292 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:24:56.517025 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513942 4292 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:24:56.517025 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513945 4292 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:24:56.517547 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513947 4292 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:24:56.517547 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513950 4292 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:24:56.517547 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513952 4292 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:24:56.517547 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513955 4292 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:24:56.517547 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513958 4292 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:24:56.517547 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513960 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:24:56.517547 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513963 4292 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:24:56.517547 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513965 4292 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:24:56.517547 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513968 4292 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:24:56.517547 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513971 4292 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:24:56.517547 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513973 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:24:56.517547 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513976 4292 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:24:56.517547 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513978 4292 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:24:56.517547 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.513982 4292 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:24:56.517547 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514483 4292 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:24:56.517547 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514490 4292 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:24:56.517547 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514493 4292 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:24:56.517547 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514496 4292 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:24:56.517547 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514499 4292 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:24:56.517547 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514502 4292 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:24:56.518032 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514505 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:24:56.518032 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514519 4292 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:24:56.518032 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514522 4292 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:24:56.518032 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514525 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:24:56.518032 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514528 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:24:56.518032 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514531 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:24:56.518032 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514533 4292 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:24:56.518032 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514536 4292 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:24:56.518032 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514539 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:24:56.518032 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514542 4292 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:24:56.518032 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514545 4292 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:24:56.518032 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514547 4292 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:24:56.518032 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514550 4292 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:24:56.518032 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514552 4292 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:24:56.518032 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514555 4292 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:24:56.518032 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514557 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:24:56.518032 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514567 4292 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:24:56.518032 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514570 4292 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:24:56.518032 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514572 4292 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:24:56.518739 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514575 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:24:56.518739 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514578 4292 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:24:56.518739 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514581 4292 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:24:56.518739 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514583 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:24:56.518739 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514586 4292 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:24:56.518739 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514588 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:24:56.518739 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514591 4292 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:24:56.518739 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514593 4292 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:24:56.518739 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514596 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:24:56.518739 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514599 4292 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:24:56.518739 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514601 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:24:56.518739 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514604 4292 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:24:56.518739 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514606 4292 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:24:56.518739 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514609 4292 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:24:56.518739 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514611 4292 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:24:56.518739 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514614 4292 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:24:56.518739 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514617 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:24:56.518739 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514619 4292 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:24:56.518739 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514622 4292 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:24:56.519537 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514624 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:24:56.519537 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514627 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:24:56.519537 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514629 4292 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:24:56.519537 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514632 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:24:56.519537 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514634 4292 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:24:56.519537 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514636 4292 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:24:56.519537 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514639 4292 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:24:56.519537 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514641 4292 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:24:56.519537 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514647 4292 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:24:56.519537 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514649 4292 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:24:56.519537 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514652 4292 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:24:56.519537 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514660 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:24:56.519537 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514665 4292 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:24:56.519537 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514668 4292 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:24:56.519537 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514672 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:24:56.519537 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514675 4292 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:24:56.519537 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514678 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:24:56.519537 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514681 4292 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:24:56.519537 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514684 4292 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:24:56.520087 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514686 4292 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:24:56.520087 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514689 4292 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:24:56.520087 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514692 4292 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:24:56.520087 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514694 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:24:56.520087 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514697 4292 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:24:56.520087 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514699 4292 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:24:56.520087 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514702 4292 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:24:56.520087 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514705 4292 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:24:56.520087 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514707 4292 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:24:56.520087 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514710 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:24:56.520087 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514712 4292 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:24:56.520087 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514715 4292 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:24:56.520087 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514717 4292 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:24:56.520087 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514720 4292 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:24:56.520087 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514722 4292 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:24:56.520087 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514724 4292 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:24:56.520087 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514727 4292 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:24:56.520087 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514729 4292 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:24:56.520087 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514732 4292 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:24:56.520087 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514734 4292 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:24:56.520584 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514737 4292 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:24:56.520584 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514741 4292 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:24:56.520584 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514744 4292 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:24:56.520584 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514746 4292 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:24:56.520584 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514749 4292 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:24:56.520584 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514757 4292 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:24:56.520584 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514760 4292 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:24:56.520584 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514763 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:24:56.520584 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514766 4292 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:24:56.520584 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514768 4292 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:24:56.520584 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514771 4292 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:24:56.520584 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514773 4292 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:24:56.520584 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514776 4292 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:24:56.520584 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514778 4292 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:24:56.520584 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514781 4292 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:24:56.520584 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514783 4292 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:24:56.520584 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514785 4292 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:24:56.520584 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514788 4292 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:24:56.520584 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514790 4292 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:24:56.520584 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514793 4292 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:24:56.521053 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514795 4292 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:24:56.521053 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514798 4292 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:24:56.521053 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.514802 4292 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:24:56.521053 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.514894 4292 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 19:24:56.521053 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.514909 4292 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 19:24:56.521053 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.514914 4292 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 19:24:56.521053 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.514919 4292 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 19:24:56.521053 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.514923 4292 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 19:24:56.521053 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.514926 4292 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 19:24:56.521053 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.514939 4292 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 19:24:56.521053 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.514944 4292 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 19:24:56.521053 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.514948 4292 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 19:24:56.521053 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.514950 4292 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 19:24:56.521053 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.514955 4292 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 19:24:56.521053 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.514959 4292 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 19:24:56.521053 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.514962 4292 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 19:24:56.521053 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.514965 4292 flags.go:64] FLAG: --cgroup-root="" Oct 30 19:24:56.521053 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.514968 4292 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 19:24:56.521053 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.514977 4292 flags.go:64] FLAG: --client-ca-file="" Oct 30 19:24:56.521053 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.514980 4292 flags.go:64] FLAG: --cloud-provider="external" Oct 30 19:24:56.521053 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.514984 4292 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 19:24:56.521053 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.514988 4292 flags.go:64] FLAG: --cluster-domain="" Oct 30 19:24:56.521053 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.514991 4292 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 19:24:56.521677 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.514994 4292 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 19:24:56.521677 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.514997 4292 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 19:24:56.521677 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515001 4292 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 19:24:56.521677 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515005 4292 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 19:24:56.521677 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515008 4292 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 19:24:56.521677 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515011 4292 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 19:24:56.521677 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515014 4292 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 19:24:56.521677 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515017 4292 flags.go:64] FLAG: --contention-profiling="false" Oct 30 19:24:56.521677 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515020 4292 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 19:24:56.521677 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515023 4292 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 19:24:56.521677 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515026 4292 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 19:24:56.521677 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515029 4292 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 19:24:56.521677 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515033 4292 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 19:24:56.521677 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515036 4292 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 19:24:56.521677 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515039 4292 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 19:24:56.521677 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515042 4292 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 19:24:56.521677 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515045 4292 flags.go:64] FLAG: --enable-server="true" Oct 30 19:24:56.521677 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515048 4292 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 19:24:56.521677 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515052 4292 flags.go:64] FLAG: --event-burst="100" Oct 30 19:24:56.521677 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515055 4292 flags.go:64] FLAG: --event-qps="50" Oct 30 19:24:56.521677 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515059 4292 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 19:24:56.521677 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515062 4292 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 19:24:56.521677 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515066 4292 flags.go:64] FLAG: --eviction-hard="" Oct 30 19:24:56.521677 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515071 4292 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 19:24:56.522267 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515074 4292 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 19:24:56.522267 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515077 4292 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 19:24:56.522267 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515080 4292 flags.go:64] FLAG: --eviction-soft="" Oct 30 19:24:56.522267 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515083 4292 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 19:24:56.522267 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515087 4292 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 19:24:56.522267 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515090 4292 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 19:24:56.522267 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515092 4292 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 19:24:56.522267 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515095 4292 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 19:24:56.522267 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515098 4292 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 19:24:56.522267 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515101 4292 flags.go:64] FLAG: --feature-gates="" Oct 30 19:24:56.522267 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515105 4292 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 19:24:56.522267 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515108 4292 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 19:24:56.522267 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515111 4292 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 19:24:56.522267 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515114 4292 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 19:24:56.522267 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515117 4292 flags.go:64] FLAG: --healthz-port="10248" Oct 30 19:24:56.522267 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515120 4292 flags.go:64] FLAG: --help="false" Oct 30 19:24:56.522267 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515123 4292 flags.go:64] FLAG: --hostname-override="ip-10-0-131-147.us-west-2.compute.internal" Oct 30 19:24:56.522267 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515126 4292 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 19:24:56.522267 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515130 4292 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 19:24:56.522267 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515133 4292 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 19:24:56.522267 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515136 4292 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 19:24:56.522267 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515140 4292 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 19:24:56.522267 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515143 4292 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 19:24:56.522871 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515145 4292 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 19:24:56.522871 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515148 4292 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 19:24:56.522871 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515151 4292 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 19:24:56.522871 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515155 4292 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 19:24:56.522871 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515158 4292 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 19:24:56.522871 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515160 4292 flags.go:64] FLAG: --kube-reserved="" Oct 30 19:24:56.522871 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515163 4292 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 19:24:56.522871 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515167 4292 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 19:24:56.522871 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515171 4292 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 19:24:56.522871 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515173 4292 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 19:24:56.522871 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515176 4292 flags.go:64] FLAG: --lock-file="" Oct 30 19:24:56.522871 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515179 4292 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 19:24:56.522871 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515182 4292 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 19:24:56.522871 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515185 4292 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 19:24:56.522871 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515197 4292 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 19:24:56.522871 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515200 4292 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 19:24:56.522871 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515203 4292 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 19:24:56.522871 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515206 4292 flags.go:64] FLAG: --logging-format="text" Oct 30 19:24:56.522871 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515209 4292 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 19:24:56.522871 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515212 4292 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 19:24:56.522871 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515215 4292 flags.go:64] FLAG: --manifest-url="" Oct 30 19:24:56.522871 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515218 4292 flags.go:64] FLAG: --manifest-url-header="" Oct 30 19:24:56.522871 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515222 4292 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 19:24:56.522871 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515225 4292 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 19:24:56.522871 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515229 4292 flags.go:64] FLAG: --max-pods="110" Oct 30 19:24:56.523475 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515232 4292 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 19:24:56.523475 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515234 4292 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 19:24:56.523475 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515237 4292 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 19:24:56.523475 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515240 4292 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 19:24:56.523475 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515243 4292 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 19:24:56.523475 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515246 4292 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 19:24:56.523475 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515249 4292 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 19:24:56.523475 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515256 4292 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 19:24:56.523475 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515259 4292 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 19:24:56.523475 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515262 4292 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 19:24:56.523475 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515265 4292 flags.go:64] FLAG: --pod-cidr="" Oct 30 19:24:56.523475 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515268 4292 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 19:24:56.523475 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515271 4292 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 19:24:56.523475 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515274 4292 flags.go:64] FLAG: --pods-per-core="0" Oct 30 19:24:56.523475 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515278 4292 flags.go:64] FLAG: --port="10250" Oct 30 19:24:56.523475 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515281 4292 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 19:24:56.523475 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515284 4292 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0c3760730cbcc881a" Oct 30 19:24:56.523475 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515287 4292 flags.go:64] FLAG: --qos-reserved="" Oct 30 19:24:56.523475 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515290 4292 flags.go:64] FLAG: --read-only-port="10255" Oct 30 19:24:56.523475 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515293 4292 flags.go:64] FLAG: --register-node="true" Oct 30 19:24:56.523475 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515296 4292 flags.go:64] FLAG: --register-with-taints="" Oct 30 19:24:56.523475 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515300 4292 flags.go:64] FLAG: --registry-burst="10" Oct 30 19:24:56.523475 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515303 4292 flags.go:64] FLAG: --registry-qps="5" Oct 30 19:24:56.523475 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515306 4292 flags.go:64] FLAG: --reserved-cpus="" Oct 30 19:24:56.523475 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515309 4292 flags.go:64] FLAG: --reserved-memory="" Oct 30 19:24:56.523475 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515313 4292 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 19:24:56.524064 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515316 4292 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 19:24:56.524064 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515319 4292 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 19:24:56.524064 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515322 4292 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 19:24:56.524064 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515325 4292 flags.go:64] FLAG: --runonce="false" Oct 30 19:24:56.524064 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515328 4292 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 19:24:56.524064 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515331 4292 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 19:24:56.524064 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515334 4292 flags.go:64] FLAG: --seccomp-default="false" Oct 30 19:24:56.524064 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515336 4292 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 19:24:56.524064 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515339 4292 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 19:24:56.524064 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515343 4292 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 19:24:56.524064 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515346 4292 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 19:24:56.524064 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515367 4292 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 19:24:56.524064 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515370 4292 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 19:24:56.524064 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515373 4292 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 19:24:56.524064 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515376 4292 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 19:24:56.524064 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515379 4292 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 19:24:56.524064 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515382 4292 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 19:24:56.524064 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515385 4292 flags.go:64] FLAG: --system-cgroups="" Oct 30 19:24:56.524064 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515388 4292 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 19:24:56.524064 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515393 4292 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 19:24:56.524064 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515398 4292 flags.go:64] FLAG: --tls-cert-file="" Oct 30 19:24:56.524064 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515401 4292 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 19:24:56.524064 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515410 4292 flags.go:64] FLAG: --tls-min-version="" Oct 30 19:24:56.524064 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515413 4292 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 19:24:56.524064 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515416 4292 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 19:24:56.524659 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515418 4292 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 19:24:56.524659 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515421 4292 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 19:24:56.524659 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515424 4292 flags.go:64] FLAG: --v="2" Oct 30 19:24:56.524659 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515428 4292 flags.go:64] FLAG: --version="false" Oct 30 19:24:56.524659 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515433 4292 flags.go:64] FLAG: --vmodule="" Oct 30 19:24:56.524659 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515436 4292 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 19:24:56.524659 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515440 4292 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 19:24:56.524659 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515605 4292 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:24:56.524659 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515609 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:24:56.524659 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515613 4292 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:24:56.524659 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515617 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:24:56.524659 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515619 4292 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:24:56.524659 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515622 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:24:56.524659 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515625 4292 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:24:56.524659 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515628 4292 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:24:56.524659 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515630 4292 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:24:56.524659 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515633 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:24:56.524659 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515636 4292 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:24:56.524659 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515638 4292 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:24:56.524659 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515641 4292 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:24:56.524659 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515644 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:24:56.525150 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515648 4292 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:24:56.525150 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515652 4292 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:24:56.525150 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515655 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:24:56.525150 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515658 4292 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:24:56.525150 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515661 4292 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:24:56.525150 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515663 4292 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:24:56.525150 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515667 4292 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:24:56.525150 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515670 4292 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:24:56.525150 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515672 4292 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:24:56.525150 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515675 4292 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:24:56.525150 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515678 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:24:56.525150 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515682 4292 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:24:56.525150 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515685 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:24:56.525150 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515688 4292 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:24:56.525150 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515690 4292 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:24:56.525150 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515693 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:24:56.525150 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515702 4292 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:24:56.525150 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515704 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:24:56.525150 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515707 4292 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:24:56.525656 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515709 4292 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:24:56.525656 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515712 4292 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:24:56.525656 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515714 4292 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:24:56.525656 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515717 4292 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:24:56.525656 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515719 4292 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:24:56.525656 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515722 4292 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:24:56.525656 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515724 4292 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:24:56.525656 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515726 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:24:56.525656 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515729 4292 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:24:56.525656 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515738 4292 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:24:56.525656 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515741 4292 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:24:56.525656 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515743 4292 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:24:56.525656 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515746 4292 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:24:56.525656 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515748 4292 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:24:56.525656 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515751 4292 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:24:56.525656 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515754 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:24:56.525656 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515756 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:24:56.525656 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515760 4292 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:24:56.525656 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515764 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:24:56.525656 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515768 4292 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:24:56.526166 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515770 4292 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:24:56.526166 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515773 4292 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:24:56.526166 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515775 4292 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:24:56.526166 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515778 4292 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:24:56.526166 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515782 4292 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:24:56.526166 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515784 4292 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:24:56.526166 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515787 4292 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:24:56.526166 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515790 4292 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:24:56.526166 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515792 4292 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:24:56.526166 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515795 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:24:56.526166 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515807 4292 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:24:56.526166 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515810 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:24:56.526166 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515812 4292 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:24:56.526166 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515815 4292 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:24:56.526166 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515817 4292 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:24:56.526166 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515820 4292 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:24:56.526166 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515822 4292 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:24:56.526166 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515825 4292 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:24:56.526166 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515827 4292 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:24:56.526166 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515830 4292 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:24:56.526675 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515832 4292 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:24:56.526675 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515835 4292 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:24:56.526675 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515837 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:24:56.526675 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515840 4292 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:24:56.526675 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515842 4292 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:24:56.526675 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515845 4292 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:24:56.526675 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515847 4292 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:24:56.526675 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515853 4292 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:24:56.526675 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515856 4292 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:24:56.526675 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515858 4292 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:24:56.526675 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515860 4292 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:24:56.526675 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515865 4292 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:24:56.526675 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515867 4292 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:24:56.526675 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515870 4292 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:24:56.526675 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515873 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:24:56.526675 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515875 4292 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:24:56.526675 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515879 4292 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:24:56.526675 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515882 4292 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:24:56.526675 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515885 4292 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:24:56.527277 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515887 4292 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:24:56.527277 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515890 4292 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:24:56.527277 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515893 4292 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:24:56.527277 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515895 4292 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:24:56.527277 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515903 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:24:56.527277 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515906 4292 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:24:56.527277 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515909 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:24:56.527277 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515911 4292 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:24:56.527277 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515914 4292 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:24:56.527277 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515916 4292 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:24:56.527277 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515919 4292 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:24:56.527277 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515921 4292 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:24:56.527277 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515924 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:24:56.527277 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515926 4292 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:24:56.527277 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.515951 4292 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:24:56.527277 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515994 4292 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:24:56.527831 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.515998 4292 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:24:56.527831 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516001 4292 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:24:56.527831 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516004 4292 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:24:56.527831 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516007 4292 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:24:56.527831 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516010 4292 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:24:56.527831 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516014 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:24:56.527831 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516017 4292 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:24:56.527831 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516019 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:24:56.527831 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516022 4292 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:24:56.527831 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516027 4292 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:24:56.527831 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516029 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:24:56.527831 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516032 4292 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:24:56.527831 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516034 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:24:56.527831 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516039 4292 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:24:56.527831 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516041 4292 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:24:56.527831 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516044 4292 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:24:56.527831 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516046 4292 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:24:56.527831 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516049 4292 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:24:56.527831 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516052 4292 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:24:56.527831 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516054 4292 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:24:56.528644 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516057 4292 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:24:56.528644 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516065 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:24:56.528644 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516068 4292 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:24:56.528644 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516071 4292 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:24:56.528644 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516073 4292 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:24:56.528644 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516076 4292 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:24:56.528644 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516078 4292 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:24:56.528644 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516081 4292 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:24:56.528644 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516084 4292 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:24:56.528644 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516086 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:24:56.528644 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516089 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:24:56.528644 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516093 4292 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:24:56.528644 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516096 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:24:56.528644 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516099 4292 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:24:56.528644 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516101 4292 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:24:56.528644 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516104 4292 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:24:56.528644 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516106 4292 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:24:56.528644 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516109 4292 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:24:56.528644 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516111 4292 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:24:56.529158 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516113 4292 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:24:56.529158 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516116 4292 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:24:56.529158 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516119 4292 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:24:56.529158 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516122 4292 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:24:56.529158 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516124 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:24:56.529158 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516126 4292 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:24:56.529158 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516130 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:24:56.529158 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516133 4292 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:24:56.529158 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516135 4292 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:24:56.529158 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516138 4292 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:24:56.529158 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516140 4292 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:24:56.529158 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516143 4292 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:24:56.529158 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516145 4292 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:24:56.529158 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516148 4292 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:24:56.529158 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516150 4292 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:24:56.529158 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516158 4292 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:24:56.529158 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516161 4292 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:24:56.529158 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516163 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:24:56.529158 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516166 4292 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:24:56.529158 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516168 4292 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:24:56.529725 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516170 4292 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:24:56.529725 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516173 4292 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:24:56.529725 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516175 4292 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:24:56.529725 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516178 4292 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:24:56.529725 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516180 4292 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:24:56.529725 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516183 4292 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:24:56.529725 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516185 4292 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:24:56.529725 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516188 4292 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:24:56.529725 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516190 4292 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:24:56.529725 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516193 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:24:56.529725 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516195 4292 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:24:56.529725 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516198 4292 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:24:56.529725 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516200 4292 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:24:56.529725 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516202 4292 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:24:56.529725 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516205 4292 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:24:56.529725 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516208 4292 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:24:56.529725 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516210 4292 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:24:56.529725 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516213 4292 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:24:56.529725 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516217 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:24:56.529725 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516220 4292 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:24:56.530219 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516222 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:24:56.530219 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516225 4292 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:24:56.530219 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516227 4292 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:24:56.530219 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516230 4292 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:24:56.530219 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516232 4292 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:24:56.530219 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516235 4292 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:24:56.530219 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516237 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:24:56.530219 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516239 4292 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:24:56.530219 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516248 4292 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:24:56.530219 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516251 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:24:56.530219 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516254 4292 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:24:56.530219 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516256 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:24:56.530219 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516258 4292 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:24:56.530219 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516261 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:24:56.530219 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516264 4292 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:24:56.530219 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516266 4292 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:24:56.530219 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516268 4292 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:24:56.530219 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516271 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:24:56.530219 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516273 4292 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:24:56.530689 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516276 4292 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:24:56.530689 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516278 4292 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:24:56.530689 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516281 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:24:56.530689 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516285 4292 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:24:56.530689 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516289 4292 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:24:56.530689 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516292 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:24:56.530689 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.516294 4292 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:24:56.530689 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.520961 4292 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 19:24:56.530689 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.520983 4292 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 19:24:56.530689 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.521000 4292 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 19:24:56.530689 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.521005 4292 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 19:24:56.530689 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521068 4292 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:24:56.530689 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521073 4292 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:24:56.530689 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521076 4292 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:24:56.530689 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521079 4292 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:24:56.530689 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521082 4292 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:24:56.530689 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521084 4292 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:24:56.530689 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521087 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:24:56.530689 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521090 4292 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:24:56.530689 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521092 4292 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:24:56.531184 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521095 4292 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:24:56.531184 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521097 4292 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:24:56.531184 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521100 4292 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:24:56.531184 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521102 4292 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:24:56.531184 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521105 4292 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:24:56.531184 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521108 4292 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:24:56.531184 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521111 4292 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:24:56.531184 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521113 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:24:56.531184 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521116 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:24:56.531184 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521118 4292 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:24:56.531184 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521121 4292 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:24:56.531184 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521124 4292 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:24:56.531184 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521126 4292 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:24:56.531184 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521131 4292 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:24:56.531184 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521134 4292 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:24:56.531184 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521138 4292 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:24:56.531184 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521140 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:24:56.531184 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521143 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:24:56.531184 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521146 4292 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:24:56.531657 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521150 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:24:56.531657 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521153 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:24:56.531657 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521156 4292 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:24:56.531657 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521159 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:24:56.531657 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521162 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:24:56.531657 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521165 4292 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:24:56.531657 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521168 4292 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:24:56.531657 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521171 4292 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:24:56.531657 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521173 4292 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:24:56.531657 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521176 4292 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:24:56.531657 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521179 4292 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:24:56.531657 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521181 4292 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:24:56.531657 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521184 4292 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:24:56.531657 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521186 4292 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:24:56.531657 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521189 4292 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:24:56.531657 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521191 4292 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:24:56.531657 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521194 4292 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:24:56.531657 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521196 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:24:56.531657 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521200 4292 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:24:56.531657 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521202 4292 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:24:56.532126 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521205 4292 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:24:56.532126 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521207 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:24:56.532126 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521210 4292 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:24:56.532126 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521212 4292 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:24:56.532126 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521215 4292 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:24:56.532126 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521217 4292 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:24:56.532126 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521220 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:24:56.532126 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521222 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:24:56.532126 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521225 4292 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:24:56.532126 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521227 4292 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:24:56.532126 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521230 4292 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:24:56.532126 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521232 4292 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:24:56.532126 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521235 4292 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:24:56.532126 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521238 4292 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:24:56.532126 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521240 4292 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:24:56.532126 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521243 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:24:56.532126 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521245 4292 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:24:56.532126 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521248 4292 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:24:56.532126 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521250 4292 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:24:56.532624 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521253 4292 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:24:56.532624 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521255 4292 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:24:56.532624 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521257 4292 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:24:56.532624 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521260 4292 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:24:56.532624 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521269 4292 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:24:56.532624 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521273 4292 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:24:56.532624 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521276 4292 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:24:56.532624 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521278 4292 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:24:56.532624 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521280 4292 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:24:56.532624 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521283 4292 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:24:56.532624 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521285 4292 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:24:56.532624 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521288 4292 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:24:56.532624 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521291 4292 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:24:56.532624 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521294 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:24:56.532624 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521296 4292 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:24:56.532624 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521299 4292 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:24:56.532624 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521320 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:24:56.532624 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521323 4292 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:24:56.532624 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521326 4292 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:24:56.532624 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521328 4292 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:24:56.533074 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521331 4292 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:24:56.533074 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521333 4292 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:24:56.533074 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521336 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:24:56.533074 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521339 4292 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:24:56.533074 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521341 4292 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:24:56.533074 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521345 4292 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:24:56.533074 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521369 4292 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:24:56.533074 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521373 4292 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:24:56.533074 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521376 4292 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:24:56.533074 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521378 4292 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:24:56.533074 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521381 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:24:56.533074 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521384 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:24:56.533074 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521386 4292 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:24:56.533074 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521389 4292 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:24:56.533074 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521392 4292 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:24:56.533074 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521394 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:24:56.533074 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521397 4292 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:24:56.533074 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521399 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:24:56.533074 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521402 4292 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:24:56.533577 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.521429 4292 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:24:56.533577 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521595 4292 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:24:56.533577 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521599 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:24:56.533577 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521602 4292 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:24:56.533577 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521605 4292 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:24:56.533577 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521609 4292 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:24:56.533577 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521612 4292 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:24:56.533577 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521615 4292 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:24:56.533577 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521618 4292 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:24:56.533577 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521620 4292 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:24:56.533577 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521623 4292 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:24:56.533577 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521625 4292 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:24:56.533577 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521628 4292 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:24:56.533577 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521630 4292 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:24:56.533577 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521633 4292 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:24:56.533577 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521636 4292 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:24:56.533951 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521638 4292 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:24:56.533951 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521641 4292 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:24:56.533951 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521643 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:24:56.533951 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521645 4292 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:24:56.533951 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521648 4292 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:24:56.533951 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521651 4292 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:24:56.533951 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521653 4292 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:24:56.533951 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521656 4292 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:24:56.533951 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521659 4292 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:24:56.533951 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521661 4292 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:24:56.533951 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521664 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:24:56.533951 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521666 4292 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:24:56.533951 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521670 4292 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:24:56.533951 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521675 4292 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:24:56.533951 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521678 4292 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:24:56.533951 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521681 4292 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:24:56.533951 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521684 4292 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:24:56.533951 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521687 4292 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:24:56.533951 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521690 4292 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:24:56.534650 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521693 4292 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:24:56.534650 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521695 4292 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:24:56.534650 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521698 4292 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:24:56.534650 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521701 4292 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:24:56.534650 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521704 4292 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:24:56.534650 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521707 4292 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:24:56.534650 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521709 4292 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:24:56.534650 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521712 4292 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:24:56.534650 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521714 4292 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:24:56.534650 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521717 4292 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:24:56.534650 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521720 4292 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:24:56.534650 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521722 4292 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:24:56.534650 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521726 4292 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:24:56.534650 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521728 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:24:56.534650 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521731 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:24:56.534650 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521734 4292 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:24:56.534650 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521736 4292 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:24:56.534650 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521739 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:24:56.534650 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521741 4292 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:24:56.534650 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521744 4292 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:24:56.535404 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521746 4292 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:24:56.535404 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521749 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:24:56.535404 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521751 4292 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:24:56.535404 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521754 4292 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:24:56.535404 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521756 4292 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:24:56.535404 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521759 4292 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:24:56.535404 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521762 4292 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:24:56.535404 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521767 4292 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:24:56.535404 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521770 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:24:56.535404 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521772 4292 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:24:56.535404 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521775 4292 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:24:56.535404 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521778 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:24:56.535404 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521780 4292 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:24:56.535404 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521783 4292 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:24:56.535404 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521786 4292 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:24:56.535404 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521788 4292 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:24:56.535404 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521792 4292 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:24:56.535404 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521797 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:24:56.535404 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521800 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:24:56.535968 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521802 4292 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:24:56.535968 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521805 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:24:56.535968 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521808 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:24:56.535968 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521811 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:24:56.535968 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521813 4292 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:24:56.535968 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521816 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:24:56.535968 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521818 4292 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:24:56.535968 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521821 4292 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:24:56.535968 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521823 4292 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:24:56.535968 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521826 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:24:56.535968 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521829 4292 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:24:56.535968 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521832 4292 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:24:56.535968 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521834 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:24:56.535968 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521837 4292 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:24:56.535968 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521839 4292 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:24:56.535968 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521842 4292 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:24:56.535968 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521844 4292 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:24:56.535968 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521847 4292 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:24:56.535968 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521849 4292 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:24:56.535968 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521852 4292 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:24:56.536636 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521856 4292 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:24:56.536636 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521859 4292 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:24:56.536636 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521862 4292 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:24:56.536636 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521864 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:24:56.536636 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521867 4292 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:24:56.536636 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521870 4292 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:24:56.536636 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521872 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:24:56.536636 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521875 4292 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:24:56.536636 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521877 4292 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:24:56.536636 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521880 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:24:56.536636 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521883 4292 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:24:56.536636 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521886 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:24:56.536636 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521888 4292 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:24:56.536636 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.521912 4292 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:24:56.536636 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521949 4292 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:24:56.537029 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521953 4292 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:24:56.537029 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521956 4292 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:24:56.537029 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521960 4292 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:24:56.537029 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521963 4292 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:24:56.537029 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521966 4292 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:24:56.537029 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521969 4292 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:24:56.537029 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521971 4292 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:24:56.537029 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521974 4292 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:24:56.537029 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521977 4292 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:24:56.537029 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521979 4292 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:24:56.537029 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521982 4292 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:24:56.537029 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521985 4292 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:24:56.537029 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521987 4292 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:24:56.537029 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521990 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:24:56.537029 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521992 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:24:56.537029 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521995 4292 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:24:56.537029 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.521998 4292 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:24:56.537029 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522001 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:24:56.537029 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522005 4292 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:24:56.537029 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522007 4292 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:24:56.537585 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522010 4292 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:24:56.537585 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522013 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:24:56.537585 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522016 4292 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:24:56.537585 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522018 4292 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:24:56.537585 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522021 4292 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:24:56.537585 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522023 4292 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:24:56.537585 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522026 4292 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:24:56.537585 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522029 4292 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:24:56.537585 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522032 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:24:56.537585 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522040 4292 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:24:56.537585 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522042 4292 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:24:56.537585 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522045 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:24:56.537585 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522048 4292 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:24:56.537585 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522050 4292 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:24:56.537585 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522054 4292 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:24:56.537585 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522057 4292 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:24:56.537585 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522062 4292 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:24:56.537585 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522066 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:24:56.537585 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522069 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:24:56.538281 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522072 4292 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:24:56.538281 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522075 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:24:56.538281 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522078 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:24:56.538281 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522080 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:24:56.538281 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522083 4292 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:24:56.538281 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522085 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:24:56.538281 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522088 4292 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:24:56.538281 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522091 4292 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:24:56.538281 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522093 4292 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:24:56.538281 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522096 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:24:56.538281 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522099 4292 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:24:56.538281 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522103 4292 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:24:56.538281 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522105 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:24:56.538281 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522108 4292 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:24:56.538281 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522111 4292 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:24:56.538281 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522114 4292 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:24:56.538281 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522116 4292 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:24:56.538281 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522119 4292 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:24:56.538281 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522121 4292 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:24:56.538281 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522124 4292 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:24:56.539153 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522126 4292 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:24:56.539153 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522129 4292 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:24:56.539153 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522132 4292 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:24:56.539153 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522135 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:24:56.539153 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522138 4292 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:24:56.539153 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522140 4292 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:24:56.539153 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522143 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:24:56.539153 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522147 4292 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:24:56.539153 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522150 4292 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:24:56.539153 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522153 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:24:56.539153 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522156 4292 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:24:56.539153 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522158 4292 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:24:56.539153 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522161 4292 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:24:56.539153 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522163 4292 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:24:56.539153 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522166 4292 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:24:56.539153 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522169 4292 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:24:56.539153 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522171 4292 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:24:56.539153 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522174 4292 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:24:56.539153 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522177 4292 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:24:56.539153 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522179 4292 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:24:56.539998 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522182 4292 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:24:56.539998 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522184 4292 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:24:56.539998 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522187 4292 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:24:56.539998 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522191 4292 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:24:56.539998 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522194 4292 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:24:56.539998 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522196 4292 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:24:56.539998 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522199 4292 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:24:56.539998 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522201 4292 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:24:56.539998 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522204 4292 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:24:56.539998 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522206 4292 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:24:56.539998 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522209 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:24:56.539998 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522212 4292 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:24:56.539998 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522214 4292 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:24:56.539998 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522217 4292 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:24:56.539998 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522219 4292 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:24:56.539998 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522222 4292 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:24:56.539998 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522225 4292 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:24:56.539998 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522227 4292 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:24:56.539998 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522230 4292 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:24:56.539998 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522234 4292 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:24:56.540863 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522239 4292 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:24:56.540863 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522243 4292 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:24:56.540863 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522246 4292 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:24:56.540863 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522249 4292 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:24:56.540863 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522252 4292 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:24:56.540863 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:56.522256 4292 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:24:56.540863 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.522331 4292 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 19:24:56.540863 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.523275 4292 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 19:24:56.540863 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.523325 4292 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 19:24:56.540863 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.523517 4292 server.go:1014] "Starting client certificate rotation" Oct 30 19:24:56.540863 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.523692 4292 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 19:24:56.540863 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.523743 4292 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 19:19:17 +0000 UTC" deadline="2028-05-13 14:14:07.075620652 +0000 UTC" Oct 30 19:24:56.540863 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.523793 4292 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13458h49m10.551829255s" Oct 30 19:24:56.540863 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.523963 4292 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 19:24:56.540863 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.525001 4292 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 19:24:56.540863 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.526438 4292 log.go:25] "Validated CRI v1 runtime API" Oct 30 19:24:56.541667 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.527815 4292 log.go:25] "Validated CRI v1 image API" Oct 30 19:24:56.541667 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.528552 4292 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 19:24:56.541667 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.529877 4292 fs.go:135] Filesystem UUIDs: map[3b1dc3af-4f73-49d0-9636-5555f06fcc95:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 a38366c6-7060-434d-b947-931174199099:/dev/nvme0n1p4] Oct 30 19:24:56.541819 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.529899 4292 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/11f7ed69235aad4fde75385d30e20e7bcddeb953f8af831aaa10956067e145f3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/11f7ed69235aad4fde75385d30e20e7bcddeb953f8af831aaa10956067e145f3/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/25b3880037d92a8f60e1c2d3ff001dd298cfefe4eab2e51530d71980d13f6825/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/25b3880037d92a8f60e1c2d3ff001dd298cfefe4eab2e51530d71980d13f6825/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/315a0f7a132b2ab04e7ac75fe697915b0a98ff6d3fdf50001fbcd4d4adaf5d2a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/315a0f7a132b2ab04e7ac75fe697915b0a98ff6d3fdf50001fbcd4d4adaf5d2a/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3cc73881ed59439fc78eb71b370e6d849851e175732e0f2187be3d1d42adaef6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3cc73881ed59439fc78eb71b370e6d849851e175732e0f2187be3d1d42adaef6/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3ff80dae5315e666f230525f1bcd98c989b845b44a07096c0563a1681be3e567/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3ff80dae5315e666f230525f1bcd98c989b845b44a07096c0563a1681be3e567/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/413c7b1cf9af624b23d5f9434258acc869486b4d718e815b2888be6ff87c381f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/413c7b1cf9af624b23d5f9434258acc869486b4d718e815b2888be6ff87c381f/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/659500fc8ef82e9b0ecf07fbac2742705e56a3583a37691572f5d52dbd2293fa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/659500fc8ef82e9b0ecf07fbac2742705e56a3583a37691572f5d52dbd2293fa/userdata/shm major:0 minor:197 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8abb9cdd72c205c969b92f61e96d8a8f8ff42513f80a6718bdb6ccf464ba5f6b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8abb9cdd72c205c969b92f61e96d8a8f8ff42513f80a6718bdb6ccf464ba5f6b/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b54d0851972c77e8e6a51e74806c7af996506eef1df90a95cd4627860359c4a1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b54d0851972c77e8e6a51e74806c7af996506eef1df90a95cd4627860359c4a1/userdata/shm major:0 minor:66 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ee9b88444d4f93ce2440c6b6ba3527ca9e6620a8bc1d508110b422462067d96f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ee9b88444d4f93ce2440c6b6ba3527ca9e6620a8bc1d508110b422462067d96f/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f04dfc935e4256d3d2231f330ea198145045d2c6f31ece3289f9f72fbc680d4f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f04dfc935e4256d3d2231f330ea198145045d2c6f31ece3289f9f72fbc680d4f/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fb567e9b5fd55e8e41d6248cb814e780ec6e5c2490b72fadbbbdcaa8b2a8de4c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fb567e9b5fd55e8e41d6248cb814e780ec6e5c2490b72fadbbbdcaa8b2a8de4c/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1165f4d9-846b-4f12-92c1-944193f28d21/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/1165f4d9-846b-4f12-92c1-944193f28d21/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1165f4d9-846b-4f12-92c1-944193f28d21/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/1165f4d9-846b-4f12-92c1-944193f28d21/volumes/kubernetes.io~empty-dir/tmp major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1165f4d9-846b-4f12-92c1-944193f28d21/volumes/kubernetes.io~projected/kube-api-access-nccr5:{mountpoint:/var/lib/kubelet/pods/1165f4d9-846b-4f12-92c1-944193f28d21/volumes/kubernetes.io~projected/kube-api-access-nccr5 major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2198fe85-6a7d-4cb3-a9e1-f804851fdb1d/volumes/kubernetes.io~projected/kube-api-access-jtmm9:{mountpoint:/var/lib/kubelet/pods/2198fe85-6a7d-4cb3-a9e1-f804851fdb1d/volumes/kubernetes.io~projected/kube-api-access-jtmm9 major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6ea94667-080f-4d20-887c-376aee648406/volumes/kubernetes.io~projected/kube-api-access-kj6h4:{mountpoint:/var/lib/kubelet/pods/6ea94667-080f-4d20-887c-376aee648406/volumes/kubernetes.io~projected/kube-api-access-kj6h4 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/83041ead-f80a-4e66-9ef2-44b25467952f/volumes/kubernetes.io~projected/kube-api-access-jbwtx:{mountpoint:/var/lib/kubelet/pods/83041ead-f80a-4e66-9ef2-44b25467952f/volumes/kubernetes.io~projected/kube-api-access-jbwtx major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a7d7990d-1d0f-41da-9dbf-1fbffe4af353/volumes/kubernetes.io~projected/kube-api-access-ts8ds:{mountpoint:/var/lib/kubelet/pods/a7d7990d-1d0f-41da-9dbf-1fbffe4af353/volumes/kubernetes.io~projected/kube-api-access-ts8ds major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/adeacb06-6b16-4597-be99-ff84d8efdadc/volumes/kubernetes.io~projected/kube-api-access-klhnw:{mountpoint:/var/lib/kubelet/pods/adeacb06-6b16-4597-be99-ff84d8efdadc/volumes/kubernetes.io~projected/kube-api-access-klhnw major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c9eecef1-2e58-4444-8a51-25b0b18715f5/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/c9eecef1-2e58-4444-8a51-25b0b18715f5/volumes/kubernetes.io~secret/agent-certs major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb4226c2-13b5-4ef8-97ba-ffd6126442f4/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/cb4226c2-13b5-4ef8-97ba-ffd6126442f4/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb4226c2-13b5-4ef8-97ba-ffd6126442f4/volumes/kubernetes.io~projected/kube-api-access-68ccj:{mountpoint:/var/lib/kubelet/pods/cb4226c2-13b5-4ef8-97ba-ffd6126442f4/volumes/kubernetes.io~projected/kube-api-access-68ccj major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb4226c2-13b5-4ef8-97ba-ffd6126442f4/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/cb4226c2-13b5-4ef8-97ba-ffd6126442f4/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1/volumes/kubernetes.io~projected/kube-api-access-ks5px:{mountpoint:/var/lib/kubelet/pods/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1/volumes/kubernetes.io~projected/kube-api-access-ks5px major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e1bb6034-1fd5-4a40-863d-e43314482f5c/volumes/kubernetes.io~projected/kube-api-access-xl5v5:{mountpoint:/var/lib/kubelet/pods/e1bb6034-1fd5-4a40-863d-e43314482f5c/volumes/kubernetes.io~projected/kube-api-access-xl5v5 major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fd873aa7-e90a-4819-8e78-ba123cd58cce/volumes/kubernetes.io~projected/kube-api-access-8t95w:{mountpoint:/var/lib/kubelet/pods/fd873aa7-e90a-4819-8e78-ba123cd58cce/volumes/kubernetes.io~projected/kube-api-access-8t95w major:0 minor:57 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/6455280cd4de624ae89c584cd5b5c97b41b58af24b29f8df13e0f2cdb17db9fb/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/7fc910b0d97498db5e83c10d9ce2299b0790dd6954540977c22e147c336cfa4c/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/b6ee371b6ecf87fbaa5cac7f5d9b105800d2b15d39079513abf6da967dcd3d77/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-110:{mountpoint:/var/lib/containers/storage/overlay/0cc8fb0b52af3a622bb32125a1771de2b95a162f830b2ae089cc69ea76f9b96f/merged major:0 minor:110 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/7eb8ce4b03946d5a186d988826c5076346748eacfd96ae9e2582ec643d059622/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/886c50e9a655e568daa4d664a3a3ae21b01465802b37a6f95a17851cc77eb035/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/3a1cd2dd3bbe331b88f8ce004ed63b962ea0aa89aab4b92be1f49eee52cd6222/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/a0c89d0da08b2c04024b337851c6543a50a733aef0059bd48fd4855ee3641c80/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/ebc34d2867c77971f9c790f3fd0199fbf772e50bf54fedc84cf6a82dd6da5cd3/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/bee2317dda96aa2206339309408799ffc514affb499416ab32a6d61271c86d63/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/58ce59cf01256d0c12457c2826b6067b10cae12887f80ed698847b75ff6dc73f/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/d1d23f871e182557ca2f6132ee403d89befedbe1ceac479bfeac6b774b98963f/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/ca881a2b241c6b01bd2b8bada34ff0a09b3102ddce04e6286c40a7e6964ee4b0/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-159:{mountpoint:/var/lib/containers/storage/overlay/aa7a27615900675976a5a64c91e31eb6f044d1010bee474e847e50e34012fcc1/merged major:0 minor:159 fsType:overlay blockSize:0} overlay_0-164:{mountpoint:/var/lib/containers/storage/overlay/30b14d6b2591c7945d90e45d9b0d8ba9a434de13fe404c790fbc5c17b90c1f9a/merged major:0 minor:164 fsType:overlay blockSize:0} overlay_0-169:{mountpoint:/var/lib/containers/storage/overlay/0a41a4fdb08b168563ccee57e02883a36b4b9d8cb3b35bb8f7d489518918d772/merged major:0 minor:169 fsType:overlay blockSize:0} overlay_0-174:{mountpoint:/var/lib/containers/storage/overlay/fee47f988f72c31da940525738324239a99edea888eb3a56b219e07314f4578e/merged major:0 minor:174 fsType:overlay blockSize:0} overlay_0-179:{mountpoint:/var/lib/containers/storage/overlay/812af307fc9726032cce284ec39ae95488ee0c4beec8f654057684d8ffb38f3d/merged major:0 minor:179 fsType:overlay blockSize:0} overlay_0-184:{mountpoint:/var/lib/containers/storage/overlay/71ded3abdc420d205356faec1f7b8f5944e7246462ba28c3aac2a948246bd4bf/merged major:0 minor:184 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/cfbcff6a3c2477cd7de38c2eaa7952e0558a0c606d382ac8066f22204d3c8979/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/538c44b43d1c7c9bd543969fa80c2a63c9f4c17fc7f158eecccfadd0d0ecaf73/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-205:{mountpoint:/var/lib/containers/storage/overlay/2a0879a9c1abcb796f8a4766c55c7a8c99649c1bf15d81ee067b27ac39aedf00/merged major:0 minor:205 fsType:overlay blockSize:0} overlay_0-207:{mountpoint:/var/lib/containers/storage/overlay/44e2884325209c62f06c86b923801a84ba48c7692311e7e81c074b4a2db6930c/merged major:0 minor:207 fsType:overlay blockSize:0} overlay_0-209:{mountpoint:/var/lib/containers/storage/overlay/eb3a2d6df9af711caa018b727d833a9bce42a0e667311a88e269b23f5f4816f2/merged major:0 minor:209 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/9d1a03389dffc2754751e3289174fe9788a1591289d8dd5c0f389044049b5801/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/3b6ce7ef63d4d36dad96f6539ff2a15bff94951782c06718591516dfc8e9384f/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/be224dcea1a5fe56e82cf0d9844d1bdb95d917e9af34b59fd1cbd2182207464b/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/5fb6c938366ca76bcce2798caa8355da06a99d1ebba944829ac60f8aef226ac7/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/2e65b71e2f171f2d5c5dd06f10442c3120f1ebd727296e5e4863d0f8fe25bb70/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/490d4d98b4022a1ccb9e50e4240f183b1b849f0620c5ccf7a14c7bd5247acb94/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/f47c69e372382c21a5034d6c02c4cc35215f06f0cba18f9d052819a9be6754a3/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/ef6fe66cbd4ecaf5b1610ab5cfad6151e12af0f6e716f6098b59019ee182d579/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/0524d9486229968d52bb62ff41ef7e8fd28d546e3c9740aefef82e363e9fc5c8/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/669702cda753c5089e1d4c50a2d34b15a81d1e21c2b5ff7709a5a3821f54cefa/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/2d3c33aca53ff1715479d97ec1c775ee8c602246ebf9165ddc768cabd791f2e8/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 19:24:56.541819 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.536641 4292 manager.go:217] Machine: {Timestamp:2026-10-30 19:24:56.536189489 +0000 UTC m=+0.054504936 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3199894 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec276b0d291c32218087472ba1636302 SystemUUID:ec276b0d-291c-3221-8087-472ba1636302 BootID:739feb6e-c75c-451d-a723-c22b7179be9c Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1165f4d9-846b-4f12-92c1-944193f28d21/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:47 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a7d7990d-1d0f-41da-9dbf-1fbffe4af353/volumes/kubernetes.io~projected/kube-api-access-ts8ds DeviceMajor:0 DeviceMinor:54 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/adeacb06-6b16-4597-be99-ff84d8efdadc/volumes/kubernetes.io~projected/kube-api-access-klhnw DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-184 DeviceMajor:0 DeviceMinor:184 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-207 DeviceMajor:0 DeviceMinor:207 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8abb9cdd72c205c969b92f61e96d8a8f8ff42513f80a6718bdb6ccf464ba5f6b/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ee9b88444d4f93ce2440c6b6ba3527ca9e6620a8bc1d508110b422462067d96f/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/1165f4d9-846b-4f12-92c1-944193f28d21/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:49 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3cc73881ed59439fc78eb71b370e6d849851e175732e0f2187be3d1d42adaef6/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/659500fc8ef82e9b0ecf07fbac2742705e56a3583a37691572f5d52dbd2293fa/userdata/shm DeviceMajor:0 DeviceMinor:197 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/fd873aa7-e90a-4819-8e78-ba123cd58cce/volumes/kubernetes.io~projected/kube-api-access-8t95w DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-174 DeviceMajor:0 DeviceMinor:174 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/1165f4d9-846b-4f12-92c1-944193f28d21/volumes/kubernetes.io~projected/kube-api-access-nccr5 DeviceMajor:0 DeviceMinor:52 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/413c7b1cf9af624b23d5f9434258acc869486b4d718e815b2888be6ff87c381f/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-164 DeviceMajor:0 DeviceMinor:164 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1/volumes/kubernetes.io~projected/kube-api-access-ks5px DeviceMajor:0 DeviceMinor:50 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/83041ead-f80a-4e66-9ef2-44b25467952f/volumes/kubernetes.io~projected/kube-api-access-jbwtx DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-205 DeviceMajor:0 DeviceMinor:205 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fb567e9b5fd55e8e41d6248cb814e780ec6e5c2490b72fadbbbdcaa8b2a8de4c/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3ff80dae5315e666f230525f1bcd98c989b845b44a07096c0563a1681be3e567/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-179 DeviceMajor:0 DeviceMinor:179 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/c9eecef1-2e58-4444-8a51-25b0b18715f5/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:48 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-110 DeviceMajor:0 DeviceMinor:110 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f04dfc935e4256d3d2231f330ea198145045d2c6f31ece3289f9f72fbc680d4f/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-159 DeviceMajor:0 DeviceMinor:159 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-209 DeviceMajor:0 DeviceMinor:209 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2198fe85-6a7d-4cb3-a9e1-f804851fdb1d/volumes/kubernetes.io~projected/kube-api-access-jtmm9 DeviceMajor:0 DeviceMinor:53 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/cb4226c2-13b5-4ef8-97ba-ffd6126442f4/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e1bb6034-1fd5-4a40-863d-e43314482f5c/volumes/kubernetes.io~projected/kube-api-access-xl5v5 DeviceMajor:0 DeviceMinor:51 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/cb4226c2-13b5-4ef8-97ba-ffd6126442f4/volumes/kubernetes.io~projected/kube-api-access-68ccj DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b54d0851972c77e8e6a51e74806c7af996506eef1df90a95cd4627860359c4a1/userdata/shm DeviceMajor:0 DeviceMinor:66 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-169 DeviceMajor:0 DeviceMinor:169 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/25b3880037d92a8f60e1c2d3ff001dd298cfefe4eab2e51530d71980d13f6825/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/11f7ed69235aad4fde75385d30e20e7bcddeb953f8af831aaa10956067e145f3/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/315a0f7a132b2ab04e7ac75fe697915b0a98ff6d3fdf50001fbcd4d4adaf5d2a/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cb4226c2-13b5-4ef8-97ba-ffd6126442f4/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/6ea94667-080f-4d20-887c-376aee648406/volumes/kubernetes.io~projected/kube-api-access-kj6h4 DeviceMajor:0 DeviceMinor:202 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:25b3880037d92a8 MacAddress:ca:47:23:01:80:48 Speed:10000 Mtu:8901} {Name:659500fc8ef82e9 MacAddress:9e:36:e0:13:f6:0f Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:f1:e1:59:05 Speed:0 Mtu:9001} {Name:br-int MacAddress:ea:4d:a0:2a:e4:90 Speed:0 Mtu:8901} {Name:ens5 MacAddress:06:ff:f1:e1:59:05 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:c6:b7:1f:be:b8:fd Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:be:f0:d4:b5:b0:3d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 19:24:56.541819 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.536849 4292 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 19:24:56.542239 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.536890 4292 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 19:24:56.542239 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.537141 4292 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 19:24:56.542239 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.537160 4292 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-147.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 19:24:56.542239 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.537281 4292 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 19:24:56.542239 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.537287 4292 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 19:24:56.542239 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.537297 4292 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 19:24:56.542239 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.537331 4292 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 19:24:56.542239 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.537445 4292 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 19:24:56.542239 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.537599 4292 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 19:24:56.542239 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.537695 4292 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 19:24:56.542239 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.537728 4292 kubelet.go:493] "Attempting to sync node with API server" Oct 30 19:24:56.542239 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.537741 4292 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 19:24:56.542239 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.537750 4292 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 19:24:56.542239 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.537760 4292 kubelet.go:405] "Adding apiserver pod source" Oct 30 19:24:56.542239 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.537769 4292 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 19:24:56.542239 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.538288 4292 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 19:24:56.542239 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.538298 4292 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 19:24:56.542239 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.538737 4292 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 19:24:56.542239 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.540448 4292 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 19:24:56.542239 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.540458 4292 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 19:24:56.542239 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.540558 4292 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 19:24:56.542239 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.540769 4292 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 19:24:56.542239 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.540795 4292 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 19:24:56.542239 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.540888 4292 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 19:24:56.542239 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.540902 4292 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 19:24:56.542239 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.540912 4292 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 19:24:56.542239 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.540917 4292 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 19:24:56.542239 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.540924 4292 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 19:24:56.542239 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.540930 4292 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 19:24:56.542239 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.540938 4292 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 19:24:56.542239 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.540944 4292 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 19:24:56.542239 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.540953 4292 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 19:24:56.542239 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.540960 4292 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 19:24:56.542239 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.540971 4292 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 19:24:56.542239 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.540985 4292 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 19:24:56.542239 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.541008 4292 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 19:24:56.542239 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.541013 4292 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 19:24:56.544380 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.544368 4292 server.go:1263] "Started kubelet" Oct 30 19:24:56.544855 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.544510 4292 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 19:24:56.544855 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.544588 4292 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 19:24:56.545021 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.544431 4292 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 19:24:56.545006 ip-10-0-131-147 systemd[1]: Started Kubernetes Kubelet. Oct 30 19:24:56.546062 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.546046 4292 server.go:317] "Adding debug handlers to kubelet server" Oct 30 19:24:56.546168 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.546142 4292 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 19:24:56.548902 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.548883 4292 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 19:24:56.549454 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.548894 4292 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 19:24:56.549454 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.548989 4292 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 19:24:56.549454 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.549004 4292 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 19:24:56.549454 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:24:56.549014 4292 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-147.us-west-2.compute.internal\" not found" Oct 30 19:24:56.549454 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.549087 4292 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:19:18 +0000 UTC" deadline="2028-03-26 09:36:36.933597477 +0000 UTC" Oct 30 19:24:56.549454 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.549117 4292 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12302h11m40.384482685s" Oct 30 19:24:56.549454 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.549167 4292 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 19:24:56.551423 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.551399 4292 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 19:24:56.551529 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.551426 4292 factory.go:55] Registering systemd factory Oct 30 19:24:56.551529 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.551436 4292 factory.go:223] Registration of the systemd container factory successfully Oct 30 19:24:56.553616 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.552049 4292 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1165f4d9-846b-4f12-92c1-944193f28d21" volumeName="kubernetes.io/projected/1165f4d9-846b-4f12-92c1-944193f28d21-kube-api-access-nccr5" seLinuxMountContext="" Oct 30 19:24:56.553616 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.552092 4292 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83041ead-f80a-4e66-9ef2-44b25467952f" volumeName="kubernetes.io/configmap/83041ead-f80a-4e66-9ef2-44b25467952f-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 19:24:56.553616 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.552106 4292 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7d7990d-1d0f-41da-9dbf-1fbffe4af353" volumeName="kubernetes.io/configmap/a7d7990d-1d0f-41da-9dbf-1fbffe4af353-serviceca" seLinuxMountContext="" Oct 30 19:24:56.553616 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.552119 4292 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="adeacb06-6b16-4597-be99-ff84d8efdadc" volumeName="kubernetes.io/projected/adeacb06-6b16-4597-be99-ff84d8efdadc-kube-api-access-klhnw" seLinuxMountContext="" Oct 30 19:24:56.553616 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.552135 4292 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9eecef1-2e58-4444-8a51-25b0b18715f5" volumeName="kubernetes.io/configmap/c9eecef1-2e58-4444-8a51-25b0b18715f5-konnectivity-ca" seLinuxMountContext="" Oct 30 19:24:56.553616 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.552145 4292 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb4226c2-13b5-4ef8-97ba-ffd6126442f4" volumeName="kubernetes.io/configmap/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-ovnkube-script-lib" seLinuxMountContext="" Oct 30 19:24:56.553616 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.552161 4292 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1" volumeName="kubernetes.io/projected/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-kube-api-access-ks5px" seLinuxMountContext="" Oct 30 19:24:56.553616 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.552173 4292 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1bb6034-1fd5-4a40-863d-e43314482f5c" volumeName="kubernetes.io/projected/e1bb6034-1fd5-4a40-863d-e43314482f5c-kube-api-access-xl5v5" seLinuxMountContext="" Oct 30 19:24:56.553616 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.552193 4292 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1165f4d9-846b-4f12-92c1-944193f28d21" volumeName="kubernetes.io/empty-dir/1165f4d9-846b-4f12-92c1-944193f28d21-etc-tuned" seLinuxMountContext="" Oct 30 19:24:56.553616 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.552204 4292 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1165f4d9-846b-4f12-92c1-944193f28d21" volumeName="kubernetes.io/empty-dir/1165f4d9-846b-4f12-92c1-944193f28d21-tmp" seLinuxMountContext="" Oct 30 19:24:56.553616 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.552215 4292 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2198fe85-6a7d-4cb3-a9e1-f804851fdb1d" volumeName="kubernetes.io/projected/2198fe85-6a7d-4cb3-a9e1-f804851fdb1d-kube-api-access-jtmm9" seLinuxMountContext="" Oct 30 19:24:56.553616 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.552231 4292 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7d7990d-1d0f-41da-9dbf-1fbffe4af353" volumeName="kubernetes.io/projected/a7d7990d-1d0f-41da-9dbf-1fbffe4af353-kube-api-access-ts8ds" seLinuxMountContext="" Oct 30 19:24:56.553616 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.552242 4292 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9eecef1-2e58-4444-8a51-25b0b18715f5" volumeName="kubernetes.io/secret/c9eecef1-2e58-4444-8a51-25b0b18715f5-agent-certs" seLinuxMountContext="" Oct 30 19:24:56.553616 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.552256 4292 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb4226c2-13b5-4ef8-97ba-ffd6126442f4" volumeName="kubernetes.io/projected/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-kube-api-access-68ccj" seLinuxMountContext="" Oct 30 19:24:56.553616 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.552285 4292 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1" volumeName="kubernetes.io/configmap/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-cni-binary-copy" seLinuxMountContext="" Oct 30 19:24:56.553616 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.552300 4292 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd873aa7-e90a-4819-8e78-ba123cd58cce" volumeName="kubernetes.io/configmap/fd873aa7-e90a-4819-8e78-ba123cd58cce-iptables-alerter-script" seLinuxMountContext="" Oct 30 19:24:56.553616 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.552312 4292 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2198fe85-6a7d-4cb3-a9e1-f804851fdb1d" volumeName="kubernetes.io/empty-dir/2198fe85-6a7d-4cb3-a9e1-f804851fdb1d-tmp-dir" seLinuxMountContext="" Oct 30 19:24:56.553616 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.552324 4292 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83041ead-f80a-4e66-9ef2-44b25467952f" volumeName="kubernetes.io/configmap/83041ead-f80a-4e66-9ef2-44b25467952f-cni-binary-copy" seLinuxMountContext="" Oct 30 19:24:56.553616 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.552385 4292 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83041ead-f80a-4e66-9ef2-44b25467952f" volumeName="kubernetes.io/configmap/83041ead-f80a-4e66-9ef2-44b25467952f-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 19:24:56.553616 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.552398 4292 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb4226c2-13b5-4ef8-97ba-ffd6126442f4" volumeName="kubernetes.io/configmap/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-ovnkube-config" seLinuxMountContext="" Oct 30 19:24:56.553616 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.552415 4292 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1" volumeName="kubernetes.io/configmap/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-multus-daemon-config" seLinuxMountContext="" Oct 30 19:24:56.553616 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.552427 4292 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87" volumeName="kubernetes.io/secret/e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87-original-pull-secret" seLinuxMountContext="" Oct 30 19:24:56.553616 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.552438 4292 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd873aa7-e90a-4819-8e78-ba123cd58cce" volumeName="kubernetes.io/projected/fd873aa7-e90a-4819-8e78-ba123cd58cce-kube-api-access-8t95w" seLinuxMountContext="" Oct 30 19:24:56.553616 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.552454 4292 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ea94667-080f-4d20-887c-376aee648406" volumeName="kubernetes.io/projected/6ea94667-080f-4d20-887c-376aee648406-kube-api-access-kj6h4" seLinuxMountContext="" Oct 30 19:24:56.553616 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.552466 4292 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83041ead-f80a-4e66-9ef2-44b25467952f" volumeName="kubernetes.io/projected/83041ead-f80a-4e66-9ef2-44b25467952f-kube-api-access-jbwtx" seLinuxMountContext="" Oct 30 19:24:56.553616 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.552483 4292 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb4226c2-13b5-4ef8-97ba-ffd6126442f4" volumeName="kubernetes.io/configmap/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-env-overrides" seLinuxMountContext="" Oct 30 19:24:56.553616 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.552495 4292 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb4226c2-13b5-4ef8-97ba-ffd6126442f4" volumeName="kubernetes.io/secret/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 19:24:56.553616 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.552506 4292 reconstruct.go:97] "Volume reconstruction finished" Oct 30 19:24:56.553616 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.552512 4292 reconciler.go:29] "Reconciler: start to sync state" Oct 30 19:24:56.553616 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.553527 4292 factory.go:153] Registering CRI-O factory Oct 30 19:24:56.553616 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.553543 4292 factory.go:223] Registration of the crio container factory successfully Oct 30 19:24:56.553616 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.553596 4292 factory.go:103] Registering Raw factory Oct 30 19:24:56.554774 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.553657 4292 manager.go:1199] Started watching for new ooms in manager Oct 30 19:24:56.554774 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.554046 4292 manager.go:319] Starting recovery of all containers Oct 30 19:24:56.563959 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.563936 4292 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 19:24:56.574874 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.574833 4292 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 19:24:56.574874 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.574873 4292 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 19:24:56.574989 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.574895 4292 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 19:24:56.574989 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:24:56.574946 4292 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 19:24:56.583661 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.583614 4292 generic.go:356] "Generic (PLEG): container finished" podID="83041ead-f80a-4e66-9ef2-44b25467952f" containerID="9a0a76baae4cb66e83e2cfc02fa75d29984cc381575f3f000f0b3c521e6180a6" exitCode=0 Oct 30 19:24:56.583661 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.583661 4292 generic.go:356] "Generic (PLEG): container finished" podID="83041ead-f80a-4e66-9ef2-44b25467952f" containerID="eb6a30a0d3ce419d6c8ebe8a9d3851f8933f97adc15d76adc584e9181d5dd4b7" exitCode=0 Oct 30 19:24:56.583772 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.583673 4292 generic.go:356] "Generic (PLEG): container finished" podID="83041ead-f80a-4e66-9ef2-44b25467952f" containerID="ab599efcac7bc5bd357389053c803c6bebfd00caebf70d46d4cd830e846ae144" exitCode=0 Oct 30 19:24:56.583772 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.583684 4292 generic.go:356] "Generic (PLEG): container finished" podID="83041ead-f80a-4e66-9ef2-44b25467952f" containerID="4fd5532f313fe8a9297265f6a23b0766134965eba498f8045bf5a8f4d160faad" exitCode=0 Oct 30 19:24:56.583772 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.583694 4292 generic.go:356] "Generic (PLEG): container finished" podID="83041ead-f80a-4e66-9ef2-44b25467952f" containerID="902ac762dd9530b7790f0c04fd948845b7fc991be2051f83050257f674dbb593" exitCode=0 Oct 30 19:24:56.583772 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.583706 4292 generic.go:356] "Generic (PLEG): container finished" podID="83041ead-f80a-4e66-9ef2-44b25467952f" containerID="48d796b1c1dd6ca68c018630e897ac3f363839b9b60b9e095a4773cc3da0f9f6" exitCode=0 Oct 30 19:24:56.596732 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.596711 4292 generic.go:356] "Generic (PLEG): container finished" podID="4cac457219e3215c4459c18e1cf7554f" containerID="9e88e0a62e2eaf21a1a01be885c2225bc9d2403f5882e512e47279d5372e8b37" exitCode=0 Oct 30 19:24:56.611505 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.611488 4292 manager.go:324] Recovery completed Oct 30 19:24:56.622901 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.622888 4292 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:24:56.624118 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.624095 4292 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-147.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:24:56.624189 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.624131 4292 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-147.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:24:56.624189 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.624142 4292 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-147.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 19:24:56.625335 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.625320 4292 cpu_manager.go:226] "Starting" policy="none" Oct 30 19:24:56.625335 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.625333 4292 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 19:24:56.625427 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.625333 4292 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:24:56.625427 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.625368 4292 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 19:24:56.625508 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.625478 4292 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 19:24:56.625508 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.625489 4292 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 19:24:56.625508 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.625505 4292 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 19:24:56.625589 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.625510 4292 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 19:24:56.625589 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.625516 4292 policy_none.go:50] "Start" Oct 30 19:24:56.625589 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.625528 4292 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 19:24:56.625589 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.625538 4292 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 19:24:56.625702 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.625665 4292 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 19:24:56.625702 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.625672 4292 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 19:24:56.625702 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.625678 4292 policy_none.go:44] "Start" Oct 30 19:24:56.626302 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.626287 4292 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-147.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:24:56.626368 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.626314 4292 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-147.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:24:56.626368 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.626323 4292 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-147.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 19:24:56.629359 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.629296 4292 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 19:24:56.632433 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.632422 4292 manager.go:341] "Starting Device Plugin manager" Oct 30 19:24:56.632484 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:24:56.632445 4292 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 19:24:56.632484 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.632453 4292 server.go:87] "Starting device plugin registration server" Oct 30 19:24:56.632746 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.632734 4292 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 19:24:56.632781 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.632749 4292 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 19:24:56.632830 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.632813 4292 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 19:24:56.632937 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.632903 4292 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 19:24:56.632937 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.632939 4292 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 19:24:56.633075 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.632944 4292 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 19:24:56.633428 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.633273 4292 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T19:24:56.632930831Z","UUID":"e77a0b6c-9d39-47e6-a99e-50919d719081","Handler":null,"Name":"","Endpoint":""} Oct 30 19:24:56.634024 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:24:56.634003 4292 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 19:24:56.634116 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:24:56.634044 4292 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-147.us-west-2.compute.internal\" not found" Oct 30 19:24:56.634754 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.634740 4292 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 19:24:56.634833 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.634759 4292 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 19:24:56.675744 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.675261 4292 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-147.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-147.us-west-2.compute.internal"] Oct 30 19:24:56.675838 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.675820 4292 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:24:56.676843 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.676826 4292 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-147.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:24:56.676922 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.676854 4292 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-147.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:24:56.676922 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.676868 4292 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-147.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 19:24:56.677076 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.677066 4292 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:24:56.677226 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.677213 4292 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:24:56.677983 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.677962 4292 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-147.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:24:56.678079 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.677994 4292 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-147.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:24:56.678079 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.678007 4292 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-147.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 19:24:56.678079 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.678018 4292 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-147.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:24:56.678079 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.678034 4292 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-147.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:24:56.678079 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.678043 4292 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-147.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 19:24:56.678311 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:24:56.678163 4292 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-147.us-west-2.compute.internal\" not found" node="ip-10-0-131-147.us-west-2.compute.internal" Oct 30 19:24:56.678311 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.678279 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-147.us-west-2.compute.internal" event={"ID":"799ed54bc75d16c836f178a66c89fbdc","Type":"ContainerStarted","Data":"d85ea22b22cdc26966be2f5e80fcb7ed7a267b9672e6aeb56764db282fbe8138"} Oct 30 19:24:56.678437 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.678324 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-147.us-west-2.compute.internal" event={"ID":"799ed54bc75d16c836f178a66c89fbdc","Type":"ContainerStarted","Data":"b54d0851972c77e8e6a51e74806c7af996506eef1df90a95cd4627860359c4a1"} Oct 30 19:24:56.678437 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.678333 4292 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:24:56.678437 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.678371 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-147.us-west-2.compute.internal" event={"ID":"4cac457219e3215c4459c18e1cf7554f","Type":"ContainerStarted","Data":"4daf739aa186f41db284b47c40526b4053dc21abda6f82c250ac344046c54ab4"} Oct 30 19:24:56.678437 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.678391 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-147.us-west-2.compute.internal" event={"ID":"4cac457219e3215c4459c18e1cf7554f","Type":"ContainerDied","Data":"9e88e0a62e2eaf21a1a01be885c2225bc9d2403f5882e512e47279d5372e8b37"} Oct 30 19:24:56.678437 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.678410 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-147.us-west-2.compute.internal" event={"ID":"4cac457219e3215c4459c18e1cf7554f","Type":"ContainerStarted","Data":"f04dfc935e4256d3d2231f330ea198145045d2c6f31ece3289f9f72fbc680d4f"} Oct 30 19:24:56.679239 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.679221 4292 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-147.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:24:56.679317 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.679249 4292 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-147.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:24:56.679317 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.679260 4292 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-147.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 19:24:56.679403 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:24:56.679391 4292 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-147.us-west-2.compute.internal\" not found" node="ip-10-0-131-147.us-west-2.compute.internal" Oct 30 19:24:56.733701 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.733676 4292 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:24:56.734788 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.734774 4292 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-147.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:24:56.734850 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.734798 4292 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-147.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:24:56.734850 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.734807 4292 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-147.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 19:24:56.734850 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.734824 4292 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-147.us-west-2.compute.internal" Oct 30 19:24:56.743499 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.743480 4292 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:24:56.788854 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.788837 4292 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:24:56.788927 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.788879 4292 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 19:24:56.788927 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.788892 4292 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:24:56.835675 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.835651 4292 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:24:56.849769 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.849748 4292 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-147.us-west-2.compute.internal" Oct 30 19:24:56.884783 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.884760 4292 nodeinfomanager.go:471] existing CSINode "ip-10-0-131-147.us-west-2.compute.internal" is owned by different node (oldNodeID="c745241f-02bf-436f-9d40-a397a42c496f", newNodeID=""), cleaning up... Oct 30 19:24:56.884873 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.884859 4292 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-131-147.us-west-2.compute.internal" Oct 30 19:24:56.884924 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.884916 4292 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-147.us-west-2.compute.internal" Oct 30 19:24:56.885478 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.885461 4292 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-147.us-west-2.compute.internal" event="NodeNotReady" Oct 30 19:24:56.885557 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.885492 4292 setters.go:546] "Node became not ready" node="ip-10-0-131-147.us-west-2.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-10-30T19:24:56Z","lastTransitionTime":"2026-10-30T19:24:56Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Oct 30 19:24:56.889756 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.889731 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/799ed54bc75d16c836f178a66c89fbdc-config\") pod \"kube-apiserver-proxy-ip-10-0-131-147.us-west-2.compute.internal\" (UID: \"799ed54bc75d16c836f178a66c89fbdc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-147.us-west-2.compute.internal" Oct 30 19:24:56.889844 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.889769 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4cac457219e3215c4459c18e1cf7554f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-147.us-west-2.compute.internal\" (UID: \"4cac457219e3215c4459c18e1cf7554f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-147.us-west-2.compute.internal" Oct 30 19:24:56.889844 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.889817 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4cac457219e3215c4459c18e1cf7554f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-147.us-west-2.compute.internal\" (UID: \"4cac457219e3215c4459c18e1cf7554f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-147.us-west-2.compute.internal" Oct 30 19:24:56.907652 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.907633 4292 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:24:56.907711 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:24:56.907669 4292 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-147.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-147.us-west-2.compute.internal" Oct 30 19:24:56.907711 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.907681 4292 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-147.us-west-2.compute.internal" Oct 30 19:24:56.963390 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.963309 4292 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:24:56.963390 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:24:56.963363 4292 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-147.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-147.us-west-2.compute.internal" Oct 30 19:24:56.990957 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.990942 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4cac457219e3215c4459c18e1cf7554f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-147.us-west-2.compute.internal\" (UID: \"4cac457219e3215c4459c18e1cf7554f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-147.us-west-2.compute.internal" Oct 30 19:24:56.991022 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.990963 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4cac457219e3215c4459c18e1cf7554f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-147.us-west-2.compute.internal\" (UID: \"4cac457219e3215c4459c18e1cf7554f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-147.us-west-2.compute.internal" Oct 30 19:24:56.991022 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.990981 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/799ed54bc75d16c836f178a66c89fbdc-config\") pod \"kube-apiserver-proxy-ip-10-0-131-147.us-west-2.compute.internal\" (UID: \"799ed54bc75d16c836f178a66c89fbdc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-147.us-west-2.compute.internal" Oct 30 19:24:56.991106 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.991034 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/799ed54bc75d16c836f178a66c89fbdc-config\") pod \"kube-apiserver-proxy-ip-10-0-131-147.us-west-2.compute.internal\" (UID: \"799ed54bc75d16c836f178a66c89fbdc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-147.us-west-2.compute.internal" Oct 30 19:24:56.991106 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.991051 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4cac457219e3215c4459c18e1cf7554f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-147.us-west-2.compute.internal\" (UID: \"4cac457219e3215c4459c18e1cf7554f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-147.us-west-2.compute.internal" Oct 30 19:24:56.991106 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:56.991058 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4cac457219e3215c4459c18e1cf7554f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-147.us-west-2.compute.internal\" (UID: \"4cac457219e3215c4459c18e1cf7554f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-147.us-west-2.compute.internal" Oct 30 19:24:57.538412 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.538371 4292 apiserver.go:52] "Watching apiserver" Oct 30 19:24:57.607224 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.607200 4292 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-147.us-west-2.compute.internal" Oct 30 19:24:57.607378 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.607306 4292 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-147.us-west-2.compute.internal" Oct 30 19:24:57.634822 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.634796 4292 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 19:24:57.635149 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.635132 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-kcr6x","openshift-cluster-node-tuning-operator/tuned-672ql","openshift-multus/network-metrics-daemon-mts99","openshift-network-operator/iptables-alerter-s5pd4","kube-system/konnectivity-agent-l66t6","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqxgl","openshift-image-registry/node-ca-dqztf","openshift-multus/multus-rffx6","kube-system/kube-apiserver-proxy-ip-10-0-131-147.us-west-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-147.us-west-2.compute.internal","openshift-multus/multus-additional-cni-plugins-x79ns","openshift-dns/node-resolver-p6vhl","openshift-network-diagnostics/network-check-target-t6m9p","openshift-ovn-kubernetes/ovnkube-node-phcbq"] Oct 30 19:24:57.636234 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.636207 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mts99" Oct 30 19:24:57.650749 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.650732 4292 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 19:24:57.663649 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.663629 4292 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:24:57.663710 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:24:57.663670 4292 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-147.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-147.us-west-2.compute.internal" Oct 30 19:24:57.664340 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.664327 4292 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:24:57.664398 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:24:57.664374 4292 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-147.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-147.us-west-2.compute.internal" Oct 30 19:24:57.686139 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.686119 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:24:57.686262 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.686247 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 19:24:57.687875 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.687854 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 19:24:57.688083 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.688058 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 19:24:57.688169 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.688113 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 19:24:57.688169 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.688119 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 19:24:57.688169 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.688116 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:24:57.688363 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.688168 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 19:24:57.688363 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.688295 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 19:24:57.688363 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.688340 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 19:24:57.688363 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.688346 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-vwkbz\"" Oct 30 19:24:57.688528 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.688363 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 19:24:57.688528 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.688390 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 19:24:57.688528 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.688451 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4g88n\"" Oct 30 19:24:57.688528 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.688477 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-lzgjh\"" Oct 30 19:24:57.688528 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.688488 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 19:24:57.688528 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.688491 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hthrr\"" Oct 30 19:24:57.688696 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.688536 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-t2d46\"" Oct 30 19:24:57.688696 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.688546 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-s4fj9\"" Oct 30 19:24:57.688890 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.688876 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 19:24:57.690736 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.690718 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 19:24:57.690823 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.690742 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 19:24:57.690873 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.690861 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 19:24:57.690937 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.690921 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 19:24:57.691076 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.691061 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 19:24:57.691143 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.691127 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 19:24:57.691221 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.691205 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 19:24:57.691300 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.691241 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 19:24:57.691300 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.691262 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 19:24:57.691300 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.691291 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 19:24:57.691453 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.691291 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 19:24:57.691453 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.691294 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 19:24:57.691453 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.691316 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 19:24:57.691453 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.691391 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 19:24:57.691453 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.691449 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 19:24:57.694762 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.694747 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/83041ead-f80a-4e66-9ef2-44b25467952f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x79ns\" (UID: \"83041ead-f80a-4e66-9ef2-44b25467952f\") " pod="openshift-multus/multus-additional-cni-plugins-x79ns" Oct 30 19:24:57.694804 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.694774 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c9eecef1-2e58-4444-8a51-25b0b18715f5-agent-certs\") pod \"konnectivity-agent-l66t6\" (UID: \"c9eecef1-2e58-4444-8a51-25b0b18715f5\") " pod="kube-system/konnectivity-agent-l66t6" Oct 30 19:24:57.694804 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.694787 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-ovnkube-config\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:57.694872 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.694803 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-cnibin\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:57.694872 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.694818 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks5px\" (UniqueName: \"kubernetes.io/projected/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-kube-api-access-ks5px\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:57.695005 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.694991 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-etc-kubernetes\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:57.695056 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.695043 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nccr5\" (UniqueName: \"kubernetes.io/projected/1165f4d9-846b-4f12-92c1-944193f28d21-kube-api-access-nccr5\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:57.695085 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.695047 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-ovnkube-config\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:57.695115 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.695092 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c9eecef1-2e58-4444-8a51-25b0b18715f5-agent-certs\") pod \"konnectivity-agent-l66t6\" (UID: \"c9eecef1-2e58-4444-8a51-25b0b18715f5\") " pod="kube-system/konnectivity-agent-l66t6" Oct 30 19:24:57.695115 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.695090 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-host-kubelet\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:57.695184 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.695133 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-host-run-netns\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:57.695184 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.695172 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1165f4d9-846b-4f12-92c1-944193f28d21-tmp\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:57.695246 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.695190 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-var-lib-openvswitch\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:57.695246 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.695210 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87-original-pull-secret\") pod \"global-pull-secret-syncer-kcr6x\" (UID: \"e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87\") " pod="kube-system/global-pull-secret-syncer-kcr6x" Oct 30 19:24:57.695316 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.695240 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-multus-cni-dir\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:57.695383 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.695314 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-host-var-lib-cni-bin\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:57.695383 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.695324 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1165f4d9-846b-4f12-92c1-944193f28d21-tmp\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:57.695383 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.695369 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-run\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:57.695474 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.695393 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c9eecef1-2e58-4444-8a51-25b0b18715f5-konnectivity-ca\") pod \"konnectivity-agent-l66t6\" (UID: \"c9eecef1-2e58-4444-8a51-25b0b18715f5\") " pod="kube-system/konnectivity-agent-l66t6" Oct 30 19:24:57.695474 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.695410 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-log-socket\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:57.695474 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.695425 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-env-overrides\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:57.695474 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.695438 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e1bb6034-1fd5-4a40-863d-e43314482f5c-etc-selinux\") pod \"aws-ebs-csi-driver-node-rqxgl\" (UID: \"e1bb6034-1fd5-4a40-863d-e43314482f5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqxgl" Oct 30 19:24:57.695474 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.695457 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e1bb6034-1fd5-4a40-863d-e43314482f5c-sys-fs\") pod \"aws-ebs-csi-driver-node-rqxgl\" (UID: \"e1bb6034-1fd5-4a40-863d-e43314482f5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqxgl" Oct 30 19:24:57.695649 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.695482 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87-original-pull-secret\") pod \"global-pull-secret-syncer-kcr6x\" (UID: \"e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87\") " pod="kube-system/global-pull-secret-syncer-kcr6x" Oct 30 19:24:57.695649 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.695509 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-cni-binary-copy\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:57.695649 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.695637 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-host-run-multus-certs\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:57.695735 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.695654 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c9eecef1-2e58-4444-8a51-25b0b18715f5-konnectivity-ca\") pod \"konnectivity-agent-l66t6\" (UID: \"c9eecef1-2e58-4444-8a51-25b0b18715f5\") " pod="kube-system/konnectivity-agent-l66t6" Oct 30 19:24:57.695735 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.695693 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-env-overrides\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:57.695735 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.695712 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-host-run-netns\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:57.695821 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.695743 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-host-run-ovn-kubernetes\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:57.695821 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.695764 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-host-cni-bin\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:57.695821 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.695788 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-ovn-node-metrics-cert\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:57.695821 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.695814 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-lib-modules\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:57.695943 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.695845 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87-kubelet-config\") pod \"global-pull-secret-syncer-kcr6x\" (UID: \"e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87\") " pod="kube-system/global-pull-secret-syncer-kcr6x" Oct 30 19:24:57.695943 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.695874 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e1bb6034-1fd5-4a40-863d-e43314482f5c-device-dir\") pod \"aws-ebs-csi-driver-node-rqxgl\" (UID: \"e1bb6034-1fd5-4a40-863d-e43314482f5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqxgl" Oct 30 19:24:57.695943 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.695881 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-cni-binary-copy\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:57.695943 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.695908 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-multus-conf-dir\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:57.696057 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.695942 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2198fe85-6a7d-4cb3-a9e1-f804851fdb1d-hosts-file\") pod \"node-resolver-p6vhl\" (UID: \"2198fe85-6a7d-4cb3-a9e1-f804851fdb1d\") " pod="openshift-dns/node-resolver-p6vhl" Oct 30 19:24:57.696057 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.695969 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtmm9\" (UniqueName: \"kubernetes.io/projected/2198fe85-6a7d-4cb3-a9e1-f804851fdb1d-kube-api-access-jtmm9\") pod \"node-resolver-p6vhl\" (UID: \"2198fe85-6a7d-4cb3-a9e1-f804851fdb1d\") " pod="openshift-dns/node-resolver-p6vhl" Oct 30 19:24:57.696057 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.696006 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-etc-systemd\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:57.696057 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.696030 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts8ds\" (UniqueName: \"kubernetes.io/projected/a7d7990d-1d0f-41da-9dbf-1fbffe4af353-kube-api-access-ts8ds\") pod \"node-ca-dqztf\" (UID: \"a7d7990d-1d0f-41da-9dbf-1fbffe4af353\") " pod="openshift-image-registry/node-ca-dqztf" Oct 30 19:24:57.696057 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.696044 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2198fe85-6a7d-4cb3-a9e1-f804851fdb1d-tmp-dir\") pod \"node-resolver-p6vhl\" (UID: \"2198fe85-6a7d-4cb3-a9e1-f804851fdb1d\") " pod="openshift-dns/node-resolver-p6vhl" Oct 30 19:24:57.696220 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.696123 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2198fe85-6a7d-4cb3-a9e1-f804851fdb1d-tmp-dir\") pod \"node-resolver-p6vhl\" (UID: \"2198fe85-6a7d-4cb3-a9e1-f804851fdb1d\") " pod="openshift-dns/node-resolver-p6vhl" Oct 30 19:24:57.696220 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.696134 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-etc-sysctl-d\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:57.696296 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.696220 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-host-slash\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:57.696296 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.696245 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-host-cni-netd\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:57.696296 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.696263 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:57.696296 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.696279 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-host\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:57.696501 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.696321 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e1bb6034-1fd5-4a40-863d-e43314482f5c-socket-dir\") pod \"aws-ebs-csi-driver-node-rqxgl\" (UID: \"e1bb6034-1fd5-4a40-863d-e43314482f5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqxgl" Oct 30 19:24:57.696501 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.696371 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/83041ead-f80a-4e66-9ef2-44b25467952f-cni-binary-copy\") pod \"multus-additional-cni-plugins-x79ns\" (UID: \"83041ead-f80a-4e66-9ef2-44b25467952f\") " pod="openshift-multus/multus-additional-cni-plugins-x79ns" Oct 30 19:24:57.696501 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.696406 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-etc-sysctl-conf\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:57.696501 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.696455 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-systemd-units\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:57.696501 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.696476 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87-dbus\") pod \"global-pull-secret-syncer-kcr6x\" (UID: \"e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87\") " pod="kube-system/global-pull-secret-syncer-kcr6x" Oct 30 19:24:57.696501 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.696491 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fd873aa7-e90a-4819-8e78-ba123cd58cce-iptables-alerter-script\") pod \"iptables-alerter-s5pd4\" (UID: \"fd873aa7-e90a-4819-8e78-ba123cd58cce\") " pod="openshift-network-operator/iptables-alerter-s5pd4" Oct 30 19:24:57.696748 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.696513 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-etc-sysconfig\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:57.696748 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.696552 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-etc-kubernetes\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:57.696748 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.696616 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-run-openvswitch\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:57.696748 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.696644 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-node-log\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:57.696748 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.696673 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68ccj\" (UniqueName: \"kubernetes.io/projected/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-kube-api-access-68ccj\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:57.696748 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.696688 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/83041ead-f80a-4e66-9ef2-44b25467952f-cni-binary-copy\") pod \"multus-additional-cni-plugins-x79ns\" (UID: \"83041ead-f80a-4e66-9ef2-44b25467952f\") " pod="openshift-multus/multus-additional-cni-plugins-x79ns" Oct 30 19:24:57.696748 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.696702 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adeacb06-6b16-4597-be99-ff84d8efdadc-metrics-certs\") pod \"network-metrics-daemon-mts99\" (UID: \"adeacb06-6b16-4597-be99-ff84d8efdadc\") " pod="openshift-multus/network-metrics-daemon-mts99" Oct 30 19:24:57.696748 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.696745 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fd873aa7-e90a-4819-8e78-ba123cd58cce-host-slash\") pod \"iptables-alerter-s5pd4\" (UID: \"fd873aa7-e90a-4819-8e78-ba123cd58cce\") " pod="openshift-network-operator/iptables-alerter-s5pd4" Oct 30 19:24:57.697082 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.696765 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fd873aa7-e90a-4819-8e78-ba123cd58cce-iptables-alerter-script\") pod \"iptables-alerter-s5pd4\" (UID: \"fd873aa7-e90a-4819-8e78-ba123cd58cce\") " pod="openshift-network-operator/iptables-alerter-s5pd4" Oct 30 19:24:57.697082 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.696772 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a7d7990d-1d0f-41da-9dbf-1fbffe4af353-host\") pod \"node-ca-dqztf\" (UID: \"a7d7990d-1d0f-41da-9dbf-1fbffe4af353\") " pod="openshift-image-registry/node-ca-dqztf" Oct 30 19:24:57.697082 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.696811 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj6h4\" (UniqueName: \"kubernetes.io/projected/6ea94667-080f-4d20-887c-376aee648406-kube-api-access-kj6h4\") pod \"network-check-target-t6m9p\" (UID: \"6ea94667-080f-4d20-887c-376aee648406\") " pod="openshift-network-diagnostics/network-check-target-t6m9p" Oct 30 19:24:57.697082 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.696839 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-ovnkube-script-lib\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:57.697082 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.696933 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-host-var-lib-kubelet\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:57.697082 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.696993 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-multus-socket-dir-parent\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:57.697082 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.697041 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-etc-modprobe-d\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:57.697082 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.697065 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1165f4d9-846b-4f12-92c1-944193f28d21-etc-tuned\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:57.697434 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.697101 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a7d7990d-1d0f-41da-9dbf-1fbffe4af353-serviceca\") pod \"node-ca-dqztf\" (UID: \"a7d7990d-1d0f-41da-9dbf-1fbffe4af353\") " pod="openshift-image-registry/node-ca-dqztf" Oct 30 19:24:57.697434 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.697123 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-ovnkube-script-lib\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:57.697434 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.697128 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1165f4d9-846b-4f12-92c1-944193f28d21-etc-tuned\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:57.697434 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.697153 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-var-lib-kubelet\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:57.697434 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.697258 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t95w\" (UniqueName: \"kubernetes.io/projected/fd873aa7-e90a-4819-8e78-ba123cd58cce-kube-api-access-8t95w\") pod \"iptables-alerter-s5pd4\" (UID: \"fd873aa7-e90a-4819-8e78-ba123cd58cce\") " pod="openshift-network-operator/iptables-alerter-s5pd4" Oct 30 19:24:57.697434 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.697305 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e1bb6034-1fd5-4a40-863d-e43314482f5c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rqxgl\" (UID: \"e1bb6034-1fd5-4a40-863d-e43314482f5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqxgl" Oct 30 19:24:57.697434 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.697396 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl5v5\" (UniqueName: \"kubernetes.io/projected/e1bb6034-1fd5-4a40-863d-e43314482f5c-kube-api-access-xl5v5\") pod \"aws-ebs-csi-driver-node-rqxgl\" (UID: \"e1bb6034-1fd5-4a40-863d-e43314482f5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqxgl" Oct 30 19:24:57.697434 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.697427 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-host-var-lib-cni-multus\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:57.697841 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.697449 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbwtx\" (UniqueName: \"kubernetes.io/projected/83041ead-f80a-4e66-9ef2-44b25467952f-kube-api-access-jbwtx\") pod \"multus-additional-cni-plugins-x79ns\" (UID: \"83041ead-f80a-4e66-9ef2-44b25467952f\") " pod="openshift-multus/multus-additional-cni-plugins-x79ns" Oct 30 19:24:57.697841 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.697537 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e1bb6034-1fd5-4a40-863d-e43314482f5c-registration-dir\") pod \"aws-ebs-csi-driver-node-rqxgl\" (UID: \"e1bb6034-1fd5-4a40-863d-e43314482f5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqxgl" Oct 30 19:24:57.697841 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.697621 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/83041ead-f80a-4e66-9ef2-44b25467952f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x79ns\" (UID: \"83041ead-f80a-4e66-9ef2-44b25467952f\") " pod="openshift-multus/multus-additional-cni-plugins-x79ns" Oct 30 19:24:57.697841 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.697645 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-sys\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:57.697841 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.697691 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-etc-openvswitch\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:57.697841 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.697738 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-run-ovn\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:57.697841 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.697764 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/83041ead-f80a-4e66-9ef2-44b25467952f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x79ns\" (UID: \"83041ead-f80a-4e66-9ef2-44b25467952f\") " pod="openshift-multus/multus-additional-cni-plugins-x79ns" Oct 30 19:24:57.697841 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.697767 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-host-run-k8s-cni-cncf-io\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:57.697841 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.697796 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-hostroot\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:57.697841 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.697825 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-multus-daemon-config\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:57.698244 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.697851 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/83041ead-f80a-4e66-9ef2-44b25467952f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x79ns\" (UID: \"83041ead-f80a-4e66-9ef2-44b25467952f\") " pod="openshift-multus/multus-additional-cni-plugins-x79ns" Oct 30 19:24:57.698244 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.697855 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a7d7990d-1d0f-41da-9dbf-1fbffe4af353-serviceca\") pod \"node-ca-dqztf\" (UID: \"a7d7990d-1d0f-41da-9dbf-1fbffe4af353\") " pod="openshift-image-registry/node-ca-dqztf" Oct 30 19:24:57.698244 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.697889 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-run-systemd\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:57.698244 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.697979 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-multus-daemon-config\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:57.698244 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.697993 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klhnw\" (UniqueName: \"kubernetes.io/projected/adeacb06-6b16-4597-be99-ff84d8efdadc-kube-api-access-klhnw\") pod \"network-metrics-daemon-mts99\" (UID: \"adeacb06-6b16-4597-be99-ff84d8efdadc\") " pod="openshift-multus/network-metrics-daemon-mts99" Oct 30 19:24:57.698244 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.698020 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-system-cni-dir\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:57.698244 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.698044 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/83041ead-f80a-4e66-9ef2-44b25467952f-system-cni-dir\") pod \"multus-additional-cni-plugins-x79ns\" (UID: \"83041ead-f80a-4e66-9ef2-44b25467952f\") " pod="openshift-multus/multus-additional-cni-plugins-x79ns" Oct 30 19:24:57.698244 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.698045 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/83041ead-f80a-4e66-9ef2-44b25467952f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x79ns\" (UID: \"83041ead-f80a-4e66-9ef2-44b25467952f\") " pod="openshift-multus/multus-additional-cni-plugins-x79ns" Oct 30 19:24:57.698244 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.698081 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/83041ead-f80a-4e66-9ef2-44b25467952f-cnibin\") pod \"multus-additional-cni-plugins-x79ns\" (UID: \"83041ead-f80a-4e66-9ef2-44b25467952f\") " pod="openshift-multus/multus-additional-cni-plugins-x79ns" Oct 30 19:24:57.755336 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.755308 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nccr5\" (UniqueName: \"kubernetes.io/projected/1165f4d9-846b-4f12-92c1-944193f28d21-kube-api-access-nccr5\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:57.756959 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.756935 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klhnw\" (UniqueName: \"kubernetes.io/projected/adeacb06-6b16-4597-be99-ff84d8efdadc-kube-api-access-klhnw\") pod \"network-metrics-daemon-mts99\" (UID: \"adeacb06-6b16-4597-be99-ff84d8efdadc\") " pod="openshift-multus/network-metrics-daemon-mts99" Oct 30 19:24:57.757061 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.756935 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t95w\" (UniqueName: \"kubernetes.io/projected/fd873aa7-e90a-4819-8e78-ba123cd58cce-kube-api-access-8t95w\") pod \"iptables-alerter-s5pd4\" (UID: \"fd873aa7-e90a-4819-8e78-ba123cd58cce\") " pod="openshift-network-operator/iptables-alerter-s5pd4" Oct 30 19:24:57.757061 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.757044 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbwtx\" (UniqueName: \"kubernetes.io/projected/83041ead-f80a-4e66-9ef2-44b25467952f-kube-api-access-jbwtx\") pod \"multus-additional-cni-plugins-x79ns\" (UID: \"83041ead-f80a-4e66-9ef2-44b25467952f\") " pod="openshift-multus/multus-additional-cni-plugins-x79ns" Oct 30 19:24:57.757273 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.757243 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl5v5\" (UniqueName: \"kubernetes.io/projected/e1bb6034-1fd5-4a40-863d-e43314482f5c-kube-api-access-xl5v5\") pod \"aws-ebs-csi-driver-node-rqxgl\" (UID: \"e1bb6034-1fd5-4a40-863d-e43314482f5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqxgl" Oct 30 19:24:57.757329 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.757273 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks5px\" (UniqueName: \"kubernetes.io/projected/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-kube-api-access-ks5px\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:57.757329 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.757286 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtmm9\" (UniqueName: \"kubernetes.io/projected/2198fe85-6a7d-4cb3-a9e1-f804851fdb1d-kube-api-access-jtmm9\") pod \"node-resolver-p6vhl\" (UID: \"2198fe85-6a7d-4cb3-a9e1-f804851fdb1d\") " pod="openshift-dns/node-resolver-p6vhl" Oct 30 19:24:57.757409 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.757393 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj6h4\" (UniqueName: \"kubernetes.io/projected/6ea94667-080f-4d20-887c-376aee648406-kube-api-access-kj6h4\") pod \"network-check-target-t6m9p\" (UID: \"6ea94667-080f-4d20-887c-376aee648406\") " pod="openshift-network-diagnostics/network-check-target-t6m9p" Oct 30 19:24:57.759211 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.759195 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts8ds\" (UniqueName: \"kubernetes.io/projected/a7d7990d-1d0f-41da-9dbf-1fbffe4af353-kube-api-access-ts8ds\") pod \"node-ca-dqztf\" (UID: \"a7d7990d-1d0f-41da-9dbf-1fbffe4af353\") " pod="openshift-image-registry/node-ca-dqztf" Oct 30 19:24:57.759263 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.759238 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68ccj\" (UniqueName: \"kubernetes.io/projected/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-kube-api-access-68ccj\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:57.792614 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.792564 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9jbqb\"" Oct 30 19:24:57.792691 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.792647 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 19:24:57.792775 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.792759 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 19:24:57.792818 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.792761 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-6ftg5\"" Oct 30 19:24:57.792818 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.792750 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6nsf6\"" Oct 30 19:24:57.792886 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.792830 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jxzqp\"" Oct 30 19:24:57.792925 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.792888 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-ndzf5\"" Oct 30 19:24:57.793097 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.793085 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6prqp\"" Oct 30 19:24:57.796217 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.796201 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-ovn-node-metrics-cert\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:57.798461 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.798448 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e1bb6034-1fd5-4a40-863d-e43314482f5c-registration-dir\") pod \"aws-ebs-csi-driver-node-rqxgl\" (UID: \"e1bb6034-1fd5-4a40-863d-e43314482f5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqxgl" Oct 30 19:24:57.798516 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.798469 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-sys\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:57.798516 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.798486 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-etc-openvswitch\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:57.798516 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.798503 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-run-ovn\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:57.798624 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.798519 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-host-run-k8s-cni-cncf-io\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:57.798624 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.798577 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-etc-openvswitch\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:57.798624 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.798587 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-run-ovn\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:57.798624 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.798595 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e1bb6034-1fd5-4a40-863d-e43314482f5c-registration-dir\") pod \"aws-ebs-csi-driver-node-rqxgl\" (UID: \"e1bb6034-1fd5-4a40-863d-e43314482f5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqxgl" Oct 30 19:24:57.798624 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.798577 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-sys\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:57.798764 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.798611 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-hostroot\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:57.798764 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.798628 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-host-run-k8s-cni-cncf-io\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:57.798764 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.798636 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-hostroot\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:57.798764 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.798651 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-run-systemd\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:57.798764 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.798671 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-system-cni-dir\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:57.798764 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.798696 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/83041ead-f80a-4e66-9ef2-44b25467952f-system-cni-dir\") pod \"multus-additional-cni-plugins-x79ns\" (UID: \"83041ead-f80a-4e66-9ef2-44b25467952f\") " pod="openshift-multus/multus-additional-cni-plugins-x79ns" Oct 30 19:24:57.798764 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.798703 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-run-systemd\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:57.798764 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.798719 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/83041ead-f80a-4e66-9ef2-44b25467952f-cnibin\") pod \"multus-additional-cni-plugins-x79ns\" (UID: \"83041ead-f80a-4e66-9ef2-44b25467952f\") " pod="openshift-multus/multus-additional-cni-plugins-x79ns" Oct 30 19:24:57.798764 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.798742 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/83041ead-f80a-4e66-9ef2-44b25467952f-system-cni-dir\") pod \"multus-additional-cni-plugins-x79ns\" (UID: \"83041ead-f80a-4e66-9ef2-44b25467952f\") " pod="openshift-multus/multus-additional-cni-plugins-x79ns" Oct 30 19:24:57.798764 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.798744 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/83041ead-f80a-4e66-9ef2-44b25467952f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x79ns\" (UID: \"83041ead-f80a-4e66-9ef2-44b25467952f\") " pod="openshift-multus/multus-additional-cni-plugins-x79ns" Oct 30 19:24:57.798764 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.798746 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-system-cni-dir\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:57.799082 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.798771 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-cnibin\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:57.799082 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.798768 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/83041ead-f80a-4e66-9ef2-44b25467952f-cnibin\") pod \"multus-additional-cni-plugins-x79ns\" (UID: \"83041ead-f80a-4e66-9ef2-44b25467952f\") " pod="openshift-multus/multus-additional-cni-plugins-x79ns" Oct 30 19:24:57.799082 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.798795 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-etc-kubernetes\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:57.799082 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.798807 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/83041ead-f80a-4e66-9ef2-44b25467952f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x79ns\" (UID: \"83041ead-f80a-4e66-9ef2-44b25467952f\") " pod="openshift-multus/multus-additional-cni-plugins-x79ns" Oct 30 19:24:57.799082 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.798819 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-host-kubelet\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:57.799082 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.798825 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-cnibin\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:57.799082 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.798850 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-host-run-netns\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:57.799082 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.798853 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-host-kubelet\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:57.799082 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.798859 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-etc-kubernetes\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:57.799082 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.798907 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-var-lib-openvswitch\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:57.799082 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.798924 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-host-run-netns\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:57.799082 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.798925 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-multus-cni-dir\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:57.799082 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.798957 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-var-lib-openvswitch\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:57.799082 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.798970 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-multus-cni-dir\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:57.799082 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.798957 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-host-var-lib-cni-bin\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:57.799082 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799011 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-run\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:57.799082 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.798986 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-host-var-lib-cni-bin\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:57.799082 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799032 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-log-socket\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:57.799082 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799052 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-run\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:57.799082 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799063 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e1bb6034-1fd5-4a40-863d-e43314482f5c-etc-selinux\") pod \"aws-ebs-csi-driver-node-rqxgl\" (UID: \"e1bb6034-1fd5-4a40-863d-e43314482f5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqxgl" Oct 30 19:24:57.799082 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799081 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-log-socket\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:57.799082 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799081 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e1bb6034-1fd5-4a40-863d-e43314482f5c-sys-fs\") pod \"aws-ebs-csi-driver-node-rqxgl\" (UID: \"e1bb6034-1fd5-4a40-863d-e43314482f5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqxgl" Oct 30 19:24:57.799931 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799119 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-host-run-multus-certs\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:57.799931 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799126 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e1bb6034-1fd5-4a40-863d-e43314482f5c-sys-fs\") pod \"aws-ebs-csi-driver-node-rqxgl\" (UID: \"e1bb6034-1fd5-4a40-863d-e43314482f5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqxgl" Oct 30 19:24:57.799931 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799133 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e1bb6034-1fd5-4a40-863d-e43314482f5c-etc-selinux\") pod \"aws-ebs-csi-driver-node-rqxgl\" (UID: \"e1bb6034-1fd5-4a40-863d-e43314482f5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqxgl" Oct 30 19:24:57.799931 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799137 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-host-run-netns\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:57.799931 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799165 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-host-run-multus-certs\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:57.799931 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799169 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-host-run-ovn-kubernetes\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:57.799931 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799194 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-host-cni-bin\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:57.799931 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799198 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-host-run-netns\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:57.799931 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799216 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-lib-modules\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:57.799931 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799226 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-host-cni-bin\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:57.799931 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799236 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-host-run-ovn-kubernetes\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:57.799931 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799239 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87-kubelet-config\") pod \"global-pull-secret-syncer-kcr6x\" (UID: \"e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87\") " pod="kube-system/global-pull-secret-syncer-kcr6x" Oct 30 19:24:57.799931 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799266 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e1bb6034-1fd5-4a40-863d-e43314482f5c-device-dir\") pod \"aws-ebs-csi-driver-node-rqxgl\" (UID: \"e1bb6034-1fd5-4a40-863d-e43314482f5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqxgl" Oct 30 19:24:57.799931 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799278 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87-kubelet-config\") pod \"global-pull-secret-syncer-kcr6x\" (UID: \"e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87\") " pod="kube-system/global-pull-secret-syncer-kcr6x" Oct 30 19:24:57.799931 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799290 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-multus-conf-dir\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:57.799931 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799298 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-lib-modules\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:57.799931 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799315 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2198fe85-6a7d-4cb3-a9e1-f804851fdb1d-hosts-file\") pod \"node-resolver-p6vhl\" (UID: \"2198fe85-6a7d-4cb3-a9e1-f804851fdb1d\") " pod="openshift-dns/node-resolver-p6vhl" Oct 30 19:24:57.799931 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799320 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e1bb6034-1fd5-4a40-863d-e43314482f5c-device-dir\") pod \"aws-ebs-csi-driver-node-rqxgl\" (UID: \"e1bb6034-1fd5-4a40-863d-e43314482f5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqxgl" Oct 30 19:24:57.799931 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799338 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-etc-systemd\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:57.799931 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799372 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-multus-conf-dir\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:57.799931 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799383 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-etc-sysctl-d\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:57.799931 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799406 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-host-slash\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:57.799931 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799409 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2198fe85-6a7d-4cb3-a9e1-f804851fdb1d-hosts-file\") pod \"node-resolver-p6vhl\" (UID: \"2198fe85-6a7d-4cb3-a9e1-f804851fdb1d\") " pod="openshift-dns/node-resolver-p6vhl" Oct 30 19:24:57.799931 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799429 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-host-cni-netd\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:57.799931 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799450 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-host-slash\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:57.799931 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799452 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:57.799931 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799469 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-host-cni-netd\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:57.799931 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799476 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-host\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:57.799931 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799486 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:57.799931 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799414 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-etc-systemd\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:57.799931 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799498 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e1bb6034-1fd5-4a40-863d-e43314482f5c-socket-dir\") pod \"aws-ebs-csi-driver-node-rqxgl\" (UID: \"e1bb6034-1fd5-4a40-863d-e43314482f5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqxgl" Oct 30 19:24:57.799931 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799512 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-etc-sysctl-d\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:57.799931 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799515 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-host\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:57.799931 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799532 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-etc-sysctl-conf\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:57.799931 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799526 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e1bb6034-1fd5-4a40-863d-e43314482f5c-socket-dir\") pod \"aws-ebs-csi-driver-node-rqxgl\" (UID: \"e1bb6034-1fd5-4a40-863d-e43314482f5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqxgl" Oct 30 19:24:57.799931 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799553 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-systemd-units\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:57.799931 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799572 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-systemd-units\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:57.799931 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799608 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87-dbus\") pod \"global-pull-secret-syncer-kcr6x\" (UID: \"e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87\") " pod="kube-system/global-pull-secret-syncer-kcr6x" Oct 30 19:24:57.799931 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799627 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-etc-sysctl-conf\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:57.799931 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799636 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-etc-sysconfig\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:57.799931 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799691 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87-dbus\") pod \"global-pull-secret-syncer-kcr6x\" (UID: \"e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87\") " pod="kube-system/global-pull-secret-syncer-kcr6x" Oct 30 19:24:57.799931 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799700 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-etc-sysconfig\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:57.799931 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799717 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-etc-kubernetes\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:57.799931 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799734 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-run-openvswitch\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:57.799931 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799752 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-node-log\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:57.799931 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799774 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-node-log\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:57.799931 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799775 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adeacb06-6b16-4597-be99-ff84d8efdadc-metrics-certs\") pod \"network-metrics-daemon-mts99\" (UID: \"adeacb06-6b16-4597-be99-ff84d8efdadc\") " pod="openshift-multus/network-metrics-daemon-mts99" Oct 30 19:24:57.799931 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799755 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-etc-kubernetes\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:57.799931 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799784 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb4226c2-13b5-4ef8-97ba-ffd6126442f4-run-openvswitch\") pod \"ovnkube-node-phcbq\" (UID: \"cb4226c2-13b5-4ef8-97ba-ffd6126442f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:57.799931 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799799 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fd873aa7-e90a-4819-8e78-ba123cd58cce-host-slash\") pod \"iptables-alerter-s5pd4\" (UID: \"fd873aa7-e90a-4819-8e78-ba123cd58cce\") " pod="openshift-network-operator/iptables-alerter-s5pd4" Oct 30 19:24:57.799931 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799816 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a7d7990d-1d0f-41da-9dbf-1fbffe4af353-host\") pod \"node-ca-dqztf\" (UID: \"a7d7990d-1d0f-41da-9dbf-1fbffe4af353\") " pod="openshift-image-registry/node-ca-dqztf" Oct 30 19:24:57.799931 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799832 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-host-var-lib-kubelet\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:57.799931 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799846 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-multus-socket-dir-parent\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:57.799931 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799858 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fd873aa7-e90a-4819-8e78-ba123cd58cce-host-slash\") pod \"iptables-alerter-s5pd4\" (UID: \"fd873aa7-e90a-4819-8e78-ba123cd58cce\") " pod="openshift-network-operator/iptables-alerter-s5pd4" Oct 30 19:24:57.799931 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799863 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-etc-modprobe-d\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:57.799931 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799868 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a7d7990d-1d0f-41da-9dbf-1fbffe4af353-host\") pod \"node-ca-dqztf\" (UID: \"a7d7990d-1d0f-41da-9dbf-1fbffe4af353\") " pod="openshift-image-registry/node-ca-dqztf" Oct 30 19:24:57.799931 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799885 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-var-lib-kubelet\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:57.799931 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799902 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e1bb6034-1fd5-4a40-863d-e43314482f5c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rqxgl\" (UID: \"e1bb6034-1fd5-4a40-863d-e43314482f5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqxgl" Oct 30 19:24:57.799931 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799910 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-multus-socket-dir-parent\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:57.799931 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799929 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-host-var-lib-cni-multus\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:57.799931 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799947 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-host-var-lib-kubelet\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:57.799931 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799960 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-etc-modprobe-d\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:57.801772 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799977 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e1bb6034-1fd5-4a40-863d-e43314482f5c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rqxgl\" (UID: \"e1bb6034-1fd5-4a40-863d-e43314482f5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqxgl" Oct 30 19:24:57.801772 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799987 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1-host-var-lib-cni-multus\") pod \"multus-rffx6\" (UID: \"d5e15534-ee0e-444b-b3ac-4a5f21c1fcc1\") " pod="openshift-multus/multus-rffx6" Oct 30 19:24:57.801772 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.799991 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1165f4d9-846b-4f12-92c1-944193f28d21-var-lib-kubelet\") pod \"tuned-672ql\" (UID: \"1165f4d9-846b-4f12-92c1-944193f28d21\") " pod="openshift-cluster-node-tuning-operator/tuned-672ql" Oct 30 19:24:57.801772 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.800193 4292 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 19:24:57.802101 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.802086 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adeacb06-6b16-4597-be99-ff84d8efdadc-metrics-certs\") pod \"network-metrics-daemon-mts99\" (UID: \"adeacb06-6b16-4597-be99-ff84d8efdadc\") " pod="openshift-multus/network-metrics-daemon-mts99" Oct 30 19:24:57.936635 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:57.936605 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mts99" Oct 30 19:24:58.193829 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:58.193760 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:58.339959 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:58.339928 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mts99"] Oct 30 19:24:58.347996 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:24:58.347961 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadeacb06_6b16_4597_be99_ff84d8efdadc.slice/crio-25c256d13ba82e42bcb9f54ed292648bef0d3f7f208df6f4f4e4f27d2694f7d8 WatchSource:0}: Error finding container 25c256d13ba82e42bcb9f54ed292648bef0d3f7f208df6f4f4e4f27d2694f7d8: Status 404 returned error can't find the container with id 25c256d13ba82e42bcb9f54ed292648bef0d3f7f208df6f4f4e4f27d2694f7d8 Oct 30 19:24:58.350199 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:58.350171 4292 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:24:58.610787 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:58.610753 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mts99" event={"ID":"adeacb06-6b16-4597-be99-ff84d8efdadc","Type":"ContainerStarted","Data":"25c256d13ba82e42bcb9f54ed292648bef0d3f7f208df6f4f4e4f27d2694f7d8"} Oct 30 19:24:59.758227 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:59.758200 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:24:59.758692 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:24:59.758417 4292 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:25:00.616163 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:00.616130 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mts99" event={"ID":"adeacb06-6b16-4597-be99-ff84d8efdadc","Type":"ContainerStarted","Data":"0efe815f5c179956642d25d3f7b326b7353ee862b2f1e1581adcaab50ba9ebc9"} Oct 30 19:25:01.620030 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:01.619997 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mts99" event={"ID":"adeacb06-6b16-4597-be99-ff84d8efdadc","Type":"ContainerStarted","Data":"4ad400b1e9f617995bb795906043780929e4c6eca283ec3cb14ab913a631c38d"} Oct 30 19:25:03.205829 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:03.205799 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-phcbq" Oct 30 19:25:06.079331 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:06.079296 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-t6m9p" Oct 30 19:25:06.954009 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:06.953976 4292 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-147.us-west-2.compute.internal" event="NodeReady" Oct 30 19:25:09.388035 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.387997 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-kwj6c"] Oct 30 19:25:09.390562 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.390530 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kwj6c" Oct 30 19:25:09.404670 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.404647 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-flg7z"] Oct 30 19:25:09.406139 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.406122 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c7579b978-jsqnr"] Oct 30 19:25:09.406244 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.406230 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-flg7z" Oct 30 19:25:09.407704 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.407682 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c7579b978-jsqnr" Oct 30 19:25:09.435146 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.435119 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-zg48z"] Oct 30 19:25:09.439669 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.439649 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-zg48z" Oct 30 19:25:09.451152 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.451132 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8x78l\"" Oct 30 19:25:09.451395 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.451370 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 19:25:09.451507 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.451392 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 19:25:09.451507 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.451440 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 19:25:09.461743 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.461714 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2rld\" (UniqueName: \"kubernetes.io/projected/3b1dddd9-2a1f-4a8a-bf37-a1e59b2a0834-kube-api-access-n2rld\") pod \"insights-runtime-extractor-kwj6c\" (UID: \"3b1dddd9-2a1f-4a8a-bf37-a1e59b2a0834\") " pod="openshift-insights/insights-runtime-extractor-kwj6c" Oct 30 19:25:09.461843 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.461754 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3b1dddd9-2a1f-4a8a-bf37-a1e59b2a0834-data-volume\") pod \"insights-runtime-extractor-kwj6c\" (UID: \"3b1dddd9-2a1f-4a8a-bf37-a1e59b2a0834\") " pod="openshift-insights/insights-runtime-extractor-kwj6c" Oct 30 19:25:09.461843 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.461798 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a94b1abe-f37a-439a-98e1-8e82038078ff-console-oauth-config\") pod \"console-6c7579b978-jsqnr\" (UID: \"a94b1abe-f37a-439a-98e1-8e82038078ff\") " pod="openshift-console/console-6c7579b978-jsqnr" Oct 30 19:25:09.461934 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.461865 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d7e50900-a49f-4316-9d27-f58cd07fda3e-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-flg7z\" (UID: \"d7e50900-a49f-4316-9d27-f58cd07fda3e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-flg7z" Oct 30 19:25:09.461934 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.461912 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3b1dddd9-2a1f-4a8a-bf37-a1e59b2a0834-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kwj6c\" (UID: \"3b1dddd9-2a1f-4a8a-bf37-a1e59b2a0834\") " pod="openshift-insights/insights-runtime-extractor-kwj6c" Oct 30 19:25:09.462043 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.461945 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a94b1abe-f37a-439a-98e1-8e82038078ff-console-config\") pod \"console-6c7579b978-jsqnr\" (UID: \"a94b1abe-f37a-439a-98e1-8e82038078ff\") " pod="openshift-console/console-6c7579b978-jsqnr" Oct 30 19:25:09.462043 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.462025 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3b1dddd9-2a1f-4a8a-bf37-a1e59b2a0834-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kwj6c\" (UID: \"3b1dddd9-2a1f-4a8a-bf37-a1e59b2a0834\") " pod="openshift-insights/insights-runtime-extractor-kwj6c" Oct 30 19:25:09.462147 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.462049 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a94b1abe-f37a-439a-98e1-8e82038078ff-service-ca\") pod \"console-6c7579b978-jsqnr\" (UID: \"a94b1abe-f37a-439a-98e1-8e82038078ff\") " pod="openshift-console/console-6c7579b978-jsqnr" Oct 30 19:25:09.462147 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.462066 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a94b1abe-f37a-439a-98e1-8e82038078ff-oauth-serving-cert\") pod \"console-6c7579b978-jsqnr\" (UID: \"a94b1abe-f37a-439a-98e1-8e82038078ff\") " pod="openshift-console/console-6c7579b978-jsqnr" Oct 30 19:25:09.462147 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.462089 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a94b1abe-f37a-439a-98e1-8e82038078ff-tmp\") pod \"console-6c7579b978-jsqnr\" (UID: \"a94b1abe-f37a-439a-98e1-8e82038078ff\") " pod="openshift-console/console-6c7579b978-jsqnr" Oct 30 19:25:09.462305 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.462162 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a94b1abe-f37a-439a-98e1-8e82038078ff-console-serving-cert\") pod \"console-6c7579b978-jsqnr\" (UID: \"a94b1abe-f37a-439a-98e1-8e82038078ff\") " pod="openshift-console/console-6c7579b978-jsqnr" Oct 30 19:25:09.462305 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.462190 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm8rq\" (UniqueName: \"kubernetes.io/projected/a94b1abe-f37a-439a-98e1-8e82038078ff-kube-api-access-fm8rq\") pod \"console-6c7579b978-jsqnr\" (UID: \"a94b1abe-f37a-439a-98e1-8e82038078ff\") " pod="openshift-console/console-6c7579b978-jsqnr" Oct 30 19:25:09.462305 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.462208 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3b1dddd9-2a1f-4a8a-bf37-a1e59b2a0834-crio-socket\") pod \"insights-runtime-extractor-kwj6c\" (UID: \"3b1dddd9-2a1f-4a8a-bf37-a1e59b2a0834\") " pod="openshift-insights/insights-runtime-extractor-kwj6c" Oct 30 19:25:09.470758 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.470732 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 19:25:09.470875 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.470766 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 30 19:25:09.490776 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.490740 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 19:25:09.490943 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.490838 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 19:25:09.490943 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.490856 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 19:25:09.490943 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.490900 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 19:25:09.491744 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.491721 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-v7wb4"] Oct 30 19:25:09.491744 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.491733 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 19:25:09.493583 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.493569 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-75znd"] Oct 30 19:25:09.493701 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.493686 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-v7wb4" Oct 30 19:25:09.495268 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.495251 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-75znd" Oct 30 19:25:09.498629 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.498611 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-76bf57bff9-zwlxg"] Oct 30 19:25:09.500465 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.500441 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76bf57bff9-zwlxg" Oct 30 19:25:09.520083 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.520061 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-fnrfj\"" Oct 30 19:25:09.520083 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.520062 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 19:25:09.520223 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.520126 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-r5mpm\"" Oct 30 19:25:09.520276 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.520265 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-w9lxj\"" Oct 30 19:25:09.520583 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.520570 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 19:25:09.539318 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.539284 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-flg7z"] Oct 30 19:25:09.539978 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.539956 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-zg48z"] Oct 30 19:25:09.541780 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.541760 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c7579b978-jsqnr"] Oct 30 19:25:09.543948 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.543917 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-f29wx\"" Oct 30 19:25:09.544240 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.544219 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 19:25:09.544478 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.544458 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 19:25:09.544608 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.544513 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 19:25:09.544711 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.544608 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 19:25:09.546298 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.546279 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 19:25:09.546436 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.546421 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-9th84\"" Oct 30 19:25:09.562587 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.562562 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a94b1abe-f37a-439a-98e1-8e82038078ff-console-config\") pod \"console-6c7579b978-jsqnr\" (UID: \"a94b1abe-f37a-439a-98e1-8e82038078ff\") " pod="openshift-console/console-6c7579b978-jsqnr" Oct 30 19:25:09.562701 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.562593 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a94b1abe-f37a-439a-98e1-8e82038078ff-oauth-serving-cert\") pod \"console-6c7579b978-jsqnr\" (UID: \"a94b1abe-f37a-439a-98e1-8e82038078ff\") " pod="openshift-console/console-6c7579b978-jsqnr" Oct 30 19:25:09.562701 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.562618 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e6cc8ab5-bbd4-4ad6-ad02-97b7efca7588-config-volume\") pod \"dns-default-v7wb4\" (UID: \"e6cc8ab5-bbd4-4ad6-ad02-97b7efca7588\") " pod="openshift-dns/dns-default-v7wb4" Oct 30 19:25:09.562701 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.562643 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e6cc8ab5-bbd4-4ad6-ad02-97b7efca7588-metrics-tls\") pod \"dns-default-v7wb4\" (UID: \"e6cc8ab5-bbd4-4ad6-ad02-97b7efca7588\") " pod="openshift-dns/dns-default-v7wb4" Oct 30 19:25:09.562701 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.562667 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7146ed33-b8b4-4a00-806a-3fec768db24a-cert\") pod \"ingress-canary-75znd\" (UID: \"7146ed33-b8b4-4a00-806a-3fec768db24a\") " pod="openshift-ingress-canary/ingress-canary-75znd" Oct 30 19:25:09.562859 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.562721 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3b1dddd9-2a1f-4a8a-bf37-a1e59b2a0834-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kwj6c\" (UID: \"3b1dddd9-2a1f-4a8a-bf37-a1e59b2a0834\") " pod="openshift-insights/insights-runtime-extractor-kwj6c" Oct 30 19:25:09.562859 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.562759 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6cc8ab5-bbd4-4ad6-ad02-97b7efca7588-tmp-dir\") pod \"dns-default-v7wb4\" (UID: \"e6cc8ab5-bbd4-4ad6-ad02-97b7efca7588\") " pod="openshift-dns/dns-default-v7wb4" Oct 30 19:25:09.562859 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.562782 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a94b1abe-f37a-439a-98e1-8e82038078ff-service-ca\") pod \"console-6c7579b978-jsqnr\" (UID: \"a94b1abe-f37a-439a-98e1-8e82038078ff\") " pod="openshift-console/console-6c7579b978-jsqnr" Oct 30 19:25:09.562859 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.562804 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a94b1abe-f37a-439a-98e1-8e82038078ff-console-serving-cert\") pod \"console-6c7579b978-jsqnr\" (UID: \"a94b1abe-f37a-439a-98e1-8e82038078ff\") " pod="openshift-console/console-6c7579b978-jsqnr" Oct 30 19:25:09.563050 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.562859 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/58267d34-0e12-4473-b64e-e02d348b6663-registry-certificates\") pod \"image-registry-76bf57bff9-zwlxg\" (UID: \"58267d34-0e12-4473-b64e-e02d348b6663\") " pod="openshift-image-registry/image-registry-76bf57bff9-zwlxg" Oct 30 19:25:09.563050 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.562898 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm8rq\" (UniqueName: \"kubernetes.io/projected/a94b1abe-f37a-439a-98e1-8e82038078ff-kube-api-access-fm8rq\") pod \"console-6c7579b978-jsqnr\" (UID: \"a94b1abe-f37a-439a-98e1-8e82038078ff\") " pod="openshift-console/console-6c7579b978-jsqnr" Oct 30 19:25:09.563050 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.562933 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pzh6\" (UniqueName: \"kubernetes.io/projected/58267d34-0e12-4473-b64e-e02d348b6663-kube-api-access-7pzh6\") pod \"image-registry-76bf57bff9-zwlxg\" (UID: \"58267d34-0e12-4473-b64e-e02d348b6663\") " pod="openshift-image-registry/image-registry-76bf57bff9-zwlxg" Oct 30 19:25:09.563050 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.562962 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/58267d34-0e12-4473-b64e-e02d348b6663-installation-pull-secrets\") pod \"image-registry-76bf57bff9-zwlxg\" (UID: \"58267d34-0e12-4473-b64e-e02d348b6663\") " pod="openshift-image-registry/image-registry-76bf57bff9-zwlxg" Oct 30 19:25:09.563050 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.562990 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltfvb\" (UniqueName: \"kubernetes.io/projected/e6cc8ab5-bbd4-4ad6-ad02-97b7efca7588-kube-api-access-ltfvb\") pod \"dns-default-v7wb4\" (UID: \"e6cc8ab5-bbd4-4ad6-ad02-97b7efca7588\") " pod="openshift-dns/dns-default-v7wb4" Oct 30 19:25:09.563050 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.563018 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28caab70-818c-447e-8974-fd3d5ef4b762-tmp\") pod \"downloads-6dd789c8b7-zg48z\" (UID: \"28caab70-818c-447e-8974-fd3d5ef4b762\") " pod="openshift-console/downloads-6dd789c8b7-zg48z" Oct 30 19:25:09.563050 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.563049 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a94b1abe-f37a-439a-98e1-8e82038078ff-console-oauth-config\") pod \"console-6c7579b978-jsqnr\" (UID: \"a94b1abe-f37a-439a-98e1-8e82038078ff\") " pod="openshift-console/console-6c7579b978-jsqnr" Oct 30 19:25:09.563387 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.563089 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3b1dddd9-2a1f-4a8a-bf37-a1e59b2a0834-data-volume\") pod \"insights-runtime-extractor-kwj6c\" (UID: \"3b1dddd9-2a1f-4a8a-bf37-a1e59b2a0834\") " pod="openshift-insights/insights-runtime-extractor-kwj6c" Oct 30 19:25:09.563387 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.563120 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kzsv\" (UniqueName: \"kubernetes.io/projected/28caab70-818c-447e-8974-fd3d5ef4b762-kube-api-access-8kzsv\") pod \"downloads-6dd789c8b7-zg48z\" (UID: \"28caab70-818c-447e-8974-fd3d5ef4b762\") " pod="openshift-console/downloads-6dd789c8b7-zg48z" Oct 30 19:25:09.563387 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.563166 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d7e50900-a49f-4316-9d27-f58cd07fda3e-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-flg7z\" (UID: \"d7e50900-a49f-4316-9d27-f58cd07fda3e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-flg7z" Oct 30 19:25:09.563387 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.563198 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3b1dddd9-2a1f-4a8a-bf37-a1e59b2a0834-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kwj6c\" (UID: \"3b1dddd9-2a1f-4a8a-bf37-a1e59b2a0834\") " pod="openshift-insights/insights-runtime-extractor-kwj6c" Oct 30 19:25:09.563387 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.563228 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/58267d34-0e12-4473-b64e-e02d348b6663-trusted-ca\") pod \"image-registry-76bf57bff9-zwlxg\" (UID: \"58267d34-0e12-4473-b64e-e02d348b6663\") " pod="openshift-image-registry/image-registry-76bf57bff9-zwlxg" Oct 30 19:25:09.563387 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.563264 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkdnb\" (UniqueName: \"kubernetes.io/projected/7146ed33-b8b4-4a00-806a-3fec768db24a-kube-api-access-dkdnb\") pod \"ingress-canary-75znd\" (UID: \"7146ed33-b8b4-4a00-806a-3fec768db24a\") " pod="openshift-ingress-canary/ingress-canary-75znd" Oct 30 19:25:09.563387 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.563281 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3b1dddd9-2a1f-4a8a-bf37-a1e59b2a0834-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kwj6c\" (UID: \"3b1dddd9-2a1f-4a8a-bf37-a1e59b2a0834\") " pod="openshift-insights/insights-runtime-extractor-kwj6c" Oct 30 19:25:09.563387 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.563291 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/58267d34-0e12-4473-b64e-e02d348b6663-bound-sa-token\") pod \"image-registry-76bf57bff9-zwlxg\" (UID: \"58267d34-0e12-4473-b64e-e02d348b6663\") " pod="openshift-image-registry/image-registry-76bf57bff9-zwlxg" Oct 30 19:25:09.563387 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.563317 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a94b1abe-f37a-439a-98e1-8e82038078ff-tmp\") pod \"console-6c7579b978-jsqnr\" (UID: \"a94b1abe-f37a-439a-98e1-8e82038078ff\") " pod="openshift-console/console-6c7579b978-jsqnr" Oct 30 19:25:09.563387 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.563344 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/58267d34-0e12-4473-b64e-e02d348b6663-image-registry-private-configuration\") pod \"image-registry-76bf57bff9-zwlxg\" (UID: \"58267d34-0e12-4473-b64e-e02d348b6663\") " pod="openshift-image-registry/image-registry-76bf57bff9-zwlxg" Oct 30 19:25:09.563882 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.563408 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58267d34-0e12-4473-b64e-e02d348b6663-registry-tls\") pod \"image-registry-76bf57bff9-zwlxg\" (UID: \"58267d34-0e12-4473-b64e-e02d348b6663\") " pod="openshift-image-registry/image-registry-76bf57bff9-zwlxg" Oct 30 19:25:09.563882 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.563438 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3b1dddd9-2a1f-4a8a-bf37-a1e59b2a0834-crio-socket\") pod \"insights-runtime-extractor-kwj6c\" (UID: \"3b1dddd9-2a1f-4a8a-bf37-a1e59b2a0834\") " pod="openshift-insights/insights-runtime-extractor-kwj6c" Oct 30 19:25:09.563882 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.563467 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2rld\" (UniqueName: \"kubernetes.io/projected/3b1dddd9-2a1f-4a8a-bf37-a1e59b2a0834-kube-api-access-n2rld\") pod \"insights-runtime-extractor-kwj6c\" (UID: \"3b1dddd9-2a1f-4a8a-bf37-a1e59b2a0834\") " pod="openshift-insights/insights-runtime-extractor-kwj6c" Oct 30 19:25:09.563882 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.563489 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/58267d34-0e12-4473-b64e-e02d348b6663-ca-trust-extracted\") pod \"image-registry-76bf57bff9-zwlxg\" (UID: \"58267d34-0e12-4473-b64e-e02d348b6663\") " pod="openshift-image-registry/image-registry-76bf57bff9-zwlxg" Oct 30 19:25:09.563882 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.563514 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3b1dddd9-2a1f-4a8a-bf37-a1e59b2a0834-data-volume\") pod \"insights-runtime-extractor-kwj6c\" (UID: \"3b1dddd9-2a1f-4a8a-bf37-a1e59b2a0834\") " pod="openshift-insights/insights-runtime-extractor-kwj6c" Oct 30 19:25:09.563882 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.563539 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a94b1abe-f37a-439a-98e1-8e82038078ff-oauth-serving-cert\") pod \"console-6c7579b978-jsqnr\" (UID: \"a94b1abe-f37a-439a-98e1-8e82038078ff\") " pod="openshift-console/console-6c7579b978-jsqnr" Oct 30 19:25:09.564190 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.563927 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3b1dddd9-2a1f-4a8a-bf37-a1e59b2a0834-crio-socket\") pod \"insights-runtime-extractor-kwj6c\" (UID: \"3b1dddd9-2a1f-4a8a-bf37-a1e59b2a0834\") " pod="openshift-insights/insights-runtime-extractor-kwj6c" Oct 30 19:25:09.564190 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.564026 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a94b1abe-f37a-439a-98e1-8e82038078ff-service-ca\") pod \"console-6c7579b978-jsqnr\" (UID: \"a94b1abe-f37a-439a-98e1-8e82038078ff\") " pod="openshift-console/console-6c7579b978-jsqnr" Oct 30 19:25:09.564512 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.564288 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a94b1abe-f37a-439a-98e1-8e82038078ff-tmp\") pod \"console-6c7579b978-jsqnr\" (UID: \"a94b1abe-f37a-439a-98e1-8e82038078ff\") " pod="openshift-console/console-6c7579b978-jsqnr" Oct 30 19:25:09.564512 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.564466 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a94b1abe-f37a-439a-98e1-8e82038078ff-console-config\") pod \"console-6c7579b978-jsqnr\" (UID: \"a94b1abe-f37a-439a-98e1-8e82038078ff\") " pod="openshift-console/console-6c7579b978-jsqnr" Oct 30 19:25:09.565498 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.565476 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a94b1abe-f37a-439a-98e1-8e82038078ff-console-serving-cert\") pod \"console-6c7579b978-jsqnr\" (UID: \"a94b1abe-f37a-439a-98e1-8e82038078ff\") " pod="openshift-console/console-6c7579b978-jsqnr" Oct 30 19:25:09.565669 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.565650 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a94b1abe-f37a-439a-98e1-8e82038078ff-console-oauth-config\") pod \"console-6c7579b978-jsqnr\" (UID: \"a94b1abe-f37a-439a-98e1-8e82038078ff\") " pod="openshift-console/console-6c7579b978-jsqnr" Oct 30 19:25:09.565944 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.565926 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3b1dddd9-2a1f-4a8a-bf37-a1e59b2a0834-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kwj6c\" (UID: \"3b1dddd9-2a1f-4a8a-bf37-a1e59b2a0834\") " pod="openshift-insights/insights-runtime-extractor-kwj6c" Oct 30 19:25:09.566018 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.566002 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d7e50900-a49f-4316-9d27-f58cd07fda3e-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-flg7z\" (UID: \"d7e50900-a49f-4316-9d27-f58cd07fda3e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-flg7z" Oct 30 19:25:09.590674 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.590650 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 19:25:09.591339 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.591319 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-75znd"] Oct 30 19:25:09.591780 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.591761 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-76bf57bff9-zwlxg"] Oct 30 19:25:09.599623 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.599599 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kwj6c"] Oct 30 19:25:09.623636 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.623603 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2rld\" (UniqueName: \"kubernetes.io/projected/3b1dddd9-2a1f-4a8a-bf37-a1e59b2a0834-kube-api-access-n2rld\") pod \"insights-runtime-extractor-kwj6c\" (UID: \"3b1dddd9-2a1f-4a8a-bf37-a1e59b2a0834\") " pod="openshift-insights/insights-runtime-extractor-kwj6c" Oct 30 19:25:09.623746 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.623675 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm8rq\" (UniqueName: \"kubernetes.io/projected/a94b1abe-f37a-439a-98e1-8e82038078ff-kube-api-access-fm8rq\") pod \"console-6c7579b978-jsqnr\" (UID: \"a94b1abe-f37a-439a-98e1-8e82038078ff\") " pod="openshift-console/console-6c7579b978-jsqnr" Oct 30 19:25:09.643747 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.643684 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 19:25:09.643747 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.643736 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 19:25:09.643883 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.643859 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 19:25:09.643917 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.643904 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-wzkg2\"" Oct 30 19:25:09.658966 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.658937 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-v7wb4"] Oct 30 19:25:09.664120 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.664098 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6cc8ab5-bbd4-4ad6-ad02-97b7efca7588-tmp-dir\") pod \"dns-default-v7wb4\" (UID: \"e6cc8ab5-bbd4-4ad6-ad02-97b7efca7588\") " pod="openshift-dns/dns-default-v7wb4" Oct 30 19:25:09.664256 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.664133 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/58267d34-0e12-4473-b64e-e02d348b6663-registry-certificates\") pod \"image-registry-76bf57bff9-zwlxg\" (UID: \"58267d34-0e12-4473-b64e-e02d348b6663\") " pod="openshift-image-registry/image-registry-76bf57bff9-zwlxg" Oct 30 19:25:09.664256 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.664154 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pzh6\" (UniqueName: \"kubernetes.io/projected/58267d34-0e12-4473-b64e-e02d348b6663-kube-api-access-7pzh6\") pod \"image-registry-76bf57bff9-zwlxg\" (UID: \"58267d34-0e12-4473-b64e-e02d348b6663\") " pod="openshift-image-registry/image-registry-76bf57bff9-zwlxg" Oct 30 19:25:09.664256 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.664177 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/58267d34-0e12-4473-b64e-e02d348b6663-installation-pull-secrets\") pod \"image-registry-76bf57bff9-zwlxg\" (UID: \"58267d34-0e12-4473-b64e-e02d348b6663\") " pod="openshift-image-registry/image-registry-76bf57bff9-zwlxg" Oct 30 19:25:09.664256 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.664201 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltfvb\" (UniqueName: \"kubernetes.io/projected/e6cc8ab5-bbd4-4ad6-ad02-97b7efca7588-kube-api-access-ltfvb\") pod \"dns-default-v7wb4\" (UID: \"e6cc8ab5-bbd4-4ad6-ad02-97b7efca7588\") " pod="openshift-dns/dns-default-v7wb4" Oct 30 19:25:09.664256 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.664226 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28caab70-818c-447e-8974-fd3d5ef4b762-tmp\") pod \"downloads-6dd789c8b7-zg48z\" (UID: \"28caab70-818c-447e-8974-fd3d5ef4b762\") " pod="openshift-console/downloads-6dd789c8b7-zg48z" Oct 30 19:25:09.664256 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.664253 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kzsv\" (UniqueName: \"kubernetes.io/projected/28caab70-818c-447e-8974-fd3d5ef4b762-kube-api-access-8kzsv\") pod \"downloads-6dd789c8b7-zg48z\" (UID: \"28caab70-818c-447e-8974-fd3d5ef4b762\") " pod="openshift-console/downloads-6dd789c8b7-zg48z" Oct 30 19:25:09.664581 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.664282 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/58267d34-0e12-4473-b64e-e02d348b6663-trusted-ca\") pod \"image-registry-76bf57bff9-zwlxg\" (UID: \"58267d34-0e12-4473-b64e-e02d348b6663\") " pod="openshift-image-registry/image-registry-76bf57bff9-zwlxg" Oct 30 19:25:09.664581 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.664310 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkdnb\" (UniqueName: \"kubernetes.io/projected/7146ed33-b8b4-4a00-806a-3fec768db24a-kube-api-access-dkdnb\") pod \"ingress-canary-75znd\" (UID: \"7146ed33-b8b4-4a00-806a-3fec768db24a\") " pod="openshift-ingress-canary/ingress-canary-75znd" Oct 30 19:25:09.664581 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.664335 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/58267d34-0e12-4473-b64e-e02d348b6663-bound-sa-token\") pod \"image-registry-76bf57bff9-zwlxg\" (UID: \"58267d34-0e12-4473-b64e-e02d348b6663\") " pod="openshift-image-registry/image-registry-76bf57bff9-zwlxg" Oct 30 19:25:09.664581 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.664379 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/58267d34-0e12-4473-b64e-e02d348b6663-image-registry-private-configuration\") pod \"image-registry-76bf57bff9-zwlxg\" (UID: \"58267d34-0e12-4473-b64e-e02d348b6663\") " pod="openshift-image-registry/image-registry-76bf57bff9-zwlxg" Oct 30 19:25:09.664581 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.664413 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58267d34-0e12-4473-b64e-e02d348b6663-registry-tls\") pod \"image-registry-76bf57bff9-zwlxg\" (UID: \"58267d34-0e12-4473-b64e-e02d348b6663\") " pod="openshift-image-registry/image-registry-76bf57bff9-zwlxg" Oct 30 19:25:09.664581 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.664446 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/58267d34-0e12-4473-b64e-e02d348b6663-ca-trust-extracted\") pod \"image-registry-76bf57bff9-zwlxg\" (UID: \"58267d34-0e12-4473-b64e-e02d348b6663\") " pod="openshift-image-registry/image-registry-76bf57bff9-zwlxg" Oct 30 19:25:09.664581 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.664475 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e6cc8ab5-bbd4-4ad6-ad02-97b7efca7588-config-volume\") pod \"dns-default-v7wb4\" (UID: \"e6cc8ab5-bbd4-4ad6-ad02-97b7efca7588\") " pod="openshift-dns/dns-default-v7wb4" Oct 30 19:25:09.664581 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.664504 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e6cc8ab5-bbd4-4ad6-ad02-97b7efca7588-metrics-tls\") pod \"dns-default-v7wb4\" (UID: \"e6cc8ab5-bbd4-4ad6-ad02-97b7efca7588\") " pod="openshift-dns/dns-default-v7wb4" Oct 30 19:25:09.664581 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.664530 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7146ed33-b8b4-4a00-806a-3fec768db24a-cert\") pod \"ingress-canary-75znd\" (UID: \"7146ed33-b8b4-4a00-806a-3fec768db24a\") " pod="openshift-ingress-canary/ingress-canary-75znd" Oct 30 19:25:09.664581 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.664539 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6cc8ab5-bbd4-4ad6-ad02-97b7efca7588-tmp-dir\") pod \"dns-default-v7wb4\" (UID: \"e6cc8ab5-bbd4-4ad6-ad02-97b7efca7588\") " pod="openshift-dns/dns-default-v7wb4" Oct 30 19:25:09.665089 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.665061 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28caab70-818c-447e-8974-fd3d5ef4b762-tmp\") pod \"downloads-6dd789c8b7-zg48z\" (UID: \"28caab70-818c-447e-8974-fd3d5ef4b762\") " pod="openshift-console/downloads-6dd789c8b7-zg48z" Oct 30 19:25:09.665666 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.665644 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/58267d34-0e12-4473-b64e-e02d348b6663-registry-certificates\") pod \"image-registry-76bf57bff9-zwlxg\" (UID: \"58267d34-0e12-4473-b64e-e02d348b6663\") " pod="openshift-image-registry/image-registry-76bf57bff9-zwlxg" Oct 30 19:25:09.666028 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.665997 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e6cc8ab5-bbd4-4ad6-ad02-97b7efca7588-config-volume\") pod \"dns-default-v7wb4\" (UID: \"e6cc8ab5-bbd4-4ad6-ad02-97b7efca7588\") " pod="openshift-dns/dns-default-v7wb4" Oct 30 19:25:09.666132 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.666069 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/58267d34-0e12-4473-b64e-e02d348b6663-ca-trust-extracted\") pod \"image-registry-76bf57bff9-zwlxg\" (UID: \"58267d34-0e12-4473-b64e-e02d348b6663\") " pod="openshift-image-registry/image-registry-76bf57bff9-zwlxg" Oct 30 19:25:09.666336 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.666305 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/58267d34-0e12-4473-b64e-e02d348b6663-trusted-ca\") pod \"image-registry-76bf57bff9-zwlxg\" (UID: \"58267d34-0e12-4473-b64e-e02d348b6663\") " pod="openshift-image-registry/image-registry-76bf57bff9-zwlxg" Oct 30 19:25:09.667338 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.667318 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7146ed33-b8b4-4a00-806a-3fec768db24a-cert\") pod \"ingress-canary-75znd\" (UID: \"7146ed33-b8b4-4a00-806a-3fec768db24a\") " pod="openshift-ingress-canary/ingress-canary-75znd" Oct 30 19:25:09.667455 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.667411 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58267d34-0e12-4473-b64e-e02d348b6663-registry-tls\") pod \"image-registry-76bf57bff9-zwlxg\" (UID: \"58267d34-0e12-4473-b64e-e02d348b6663\") " pod="openshift-image-registry/image-registry-76bf57bff9-zwlxg" Oct 30 19:25:09.667497 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.667480 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/58267d34-0e12-4473-b64e-e02d348b6663-installation-pull-secrets\") pod \"image-registry-76bf57bff9-zwlxg\" (UID: \"58267d34-0e12-4473-b64e-e02d348b6663\") " pod="openshift-image-registry/image-registry-76bf57bff9-zwlxg" Oct 30 19:25:09.667866 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.667844 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/58267d34-0e12-4473-b64e-e02d348b6663-image-registry-private-configuration\") pod \"image-registry-76bf57bff9-zwlxg\" (UID: \"58267d34-0e12-4473-b64e-e02d348b6663\") " pod="openshift-image-registry/image-registry-76bf57bff9-zwlxg" Oct 30 19:25:09.668070 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.668052 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e6cc8ab5-bbd4-4ad6-ad02-97b7efca7588-metrics-tls\") pod \"dns-default-v7wb4\" (UID: \"e6cc8ab5-bbd4-4ad6-ad02-97b7efca7588\") " pod="openshift-dns/dns-default-v7wb4" Oct 30 19:25:09.702810 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.702789 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kwj6c" Oct 30 19:25:09.716317 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.716293 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-flg7z" Oct 30 19:25:09.721511 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.721488 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c7579b978-jsqnr" Oct 30 19:25:09.742271 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.742242 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/58267d34-0e12-4473-b64e-e02d348b6663-bound-sa-token\") pod \"image-registry-76bf57bff9-zwlxg\" (UID: \"58267d34-0e12-4473-b64e-e02d348b6663\") " pod="openshift-image-registry/image-registry-76bf57bff9-zwlxg" Oct 30 19:25:09.742572 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.742546 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kzsv\" (UniqueName: \"kubernetes.io/projected/28caab70-818c-447e-8974-fd3d5ef4b762-kube-api-access-8kzsv\") pod \"downloads-6dd789c8b7-zg48z\" (UID: \"28caab70-818c-447e-8974-fd3d5ef4b762\") " pod="openshift-console/downloads-6dd789c8b7-zg48z" Oct 30 19:25:09.742622 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.742575 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltfvb\" (UniqueName: \"kubernetes.io/projected/e6cc8ab5-bbd4-4ad6-ad02-97b7efca7588-kube-api-access-ltfvb\") pod \"dns-default-v7wb4\" (UID: \"e6cc8ab5-bbd4-4ad6-ad02-97b7efca7588\") " pod="openshift-dns/dns-default-v7wb4" Oct 30 19:25:09.743118 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.743099 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pzh6\" (UniqueName: \"kubernetes.io/projected/58267d34-0e12-4473-b64e-e02d348b6663-kube-api-access-7pzh6\") pod \"image-registry-76bf57bff9-zwlxg\" (UID: \"58267d34-0e12-4473-b64e-e02d348b6663\") " pod="openshift-image-registry/image-registry-76bf57bff9-zwlxg" Oct 30 19:25:09.743802 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.743782 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkdnb\" (UniqueName: \"kubernetes.io/projected/7146ed33-b8b4-4a00-806a-3fec768db24a-kube-api-access-dkdnb\") pod \"ingress-canary-75znd\" (UID: \"7146ed33-b8b4-4a00-806a-3fec768db24a\") " pod="openshift-ingress-canary/ingress-canary-75znd" Oct 30 19:25:09.751541 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.750993 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-zg48z" Oct 30 19:25:09.805788 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.805760 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-v7wb4" Oct 30 19:25:09.810933 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.810914 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-75znd" Oct 30 19:25:09.815893 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:09.815873 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76bf57bff9-zwlxg" Oct 30 19:25:10.075889 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:10.075856 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kwj6c"] Oct 30 19:25:10.082894 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:25:10.082866 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b1dddd9_2a1f_4a8a_bf37_a1e59b2a0834.slice/crio-01a4cdfb3f644d1caad202692cd88bc0e3c2a502265a764c69d89414e0ed0f31 WatchSource:0}: Error finding container 01a4cdfb3f644d1caad202692cd88bc0e3c2a502265a764c69d89414e0ed0f31: Status 404 returned error can't find the container with id 01a4cdfb3f644d1caad202692cd88bc0e3c2a502265a764c69d89414e0ed0f31 Oct 30 19:25:10.103952 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:10.103929 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-flg7z"] Oct 30 19:25:10.112770 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:25:10.112728 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7e50900_a49f_4316_9d27_f58cd07fda3e.slice/crio-1dabde73e3ad88511c47ac8c1db238f5fe6b65711ddc91ef58487a8a5bed71eb WatchSource:0}: Error finding container 1dabde73e3ad88511c47ac8c1db238f5fe6b65711ddc91ef58487a8a5bed71eb: Status 404 returned error can't find the container with id 1dabde73e3ad88511c47ac8c1db238f5fe6b65711ddc91ef58487a8a5bed71eb Oct 30 19:25:10.128407 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:10.128385 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c7579b978-jsqnr"] Oct 30 19:25:10.135537 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:25:10.135508 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda94b1abe_f37a_439a_98e1_8e82038078ff.slice/crio-e46a2d97c071fa0f7c1447c2b463899253d9a40e9b8b7ca48f265a0fbca63d26 WatchSource:0}: Error finding container e46a2d97c071fa0f7c1447c2b463899253d9a40e9b8b7ca48f265a0fbca63d26: Status 404 returned error can't find the container with id e46a2d97c071fa0f7c1447c2b463899253d9a40e9b8b7ca48f265a0fbca63d26 Oct 30 19:25:10.149618 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:10.149560 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-zg48z"] Oct 30 19:25:10.158908 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:25:10.158887 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28caab70_818c_447e_8974_fd3d5ef4b762.slice/crio-d609012e06399a4b9c079dc5c98eb73d66fc7ef70baf18ad2dc3141dbb8c6a9f WatchSource:0}: Error finding container d609012e06399a4b9c079dc5c98eb73d66fc7ef70baf18ad2dc3141dbb8c6a9f: Status 404 returned error can't find the container with id d609012e06399a4b9c079dc5c98eb73d66fc7ef70baf18ad2dc3141dbb8c6a9f Oct 30 19:25:10.213251 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:10.213214 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-75znd"] Oct 30 19:25:10.217062 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:10.217028 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-v7wb4"] Oct 30 19:25:10.221981 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:25:10.221952 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7146ed33_b8b4_4a00_806a_3fec768db24a.slice/crio-57d4224e8c972714597b3e2137ea1a054c3fce8facf32916e387e89074a7a415 WatchSource:0}: Error finding container 57d4224e8c972714597b3e2137ea1a054c3fce8facf32916e387e89074a7a415: Status 404 returned error can't find the container with id 57d4224e8c972714597b3e2137ea1a054c3fce8facf32916e387e89074a7a415 Oct 30 19:25:10.224717 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:25:10.224691 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6cc8ab5_bbd4_4ad6_ad02_97b7efca7588.slice/crio-e807dc3bc38251c7ac4d8c04813265c5066a2f9db9e3d697e0769e7f48203ff4 WatchSource:0}: Error finding container e807dc3bc38251c7ac4d8c04813265c5066a2f9db9e3d697e0769e7f48203ff4: Status 404 returned error can't find the container with id e807dc3bc38251c7ac4d8c04813265c5066a2f9db9e3d697e0769e7f48203ff4 Oct 30 19:25:10.358914 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:10.358876 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-76bf57bff9-zwlxg"] Oct 30 19:25:10.366489 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:25:10.366464 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58267d34_0e12_4473_b64e_e02d348b6663.slice/crio-0037e0b1844d1e4246f73c8baf57efef62468038b21c18a1510fdab7312a4916 WatchSource:0}: Error finding container 0037e0b1844d1e4246f73c8baf57efef62468038b21c18a1510fdab7312a4916: Status 404 returned error can't find the container with id 0037e0b1844d1e4246f73c8baf57efef62468038b21c18a1510fdab7312a4916 Oct 30 19:25:10.642903 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:10.642848 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76bf57bff9-zwlxg" event={"ID":"58267d34-0e12-4473-b64e-e02d348b6663","Type":"ContainerStarted","Data":"b958de3ab33c3ab9cfcd0faf1edd3bcdf79b20263a97002c8a6fa28a475adfbb"} Oct 30 19:25:10.643322 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:10.642909 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76bf57bff9-zwlxg" event={"ID":"58267d34-0e12-4473-b64e-e02d348b6663","Type":"ContainerStarted","Data":"0037e0b1844d1e4246f73c8baf57efef62468038b21c18a1510fdab7312a4916"} Oct 30 19:25:10.643322 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:10.642968 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-76bf57bff9-zwlxg" Oct 30 19:25:10.644181 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:10.644156 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-zg48z" event={"ID":"28caab70-818c-447e-8974-fd3d5ef4b762","Type":"ContainerStarted","Data":"d609012e06399a4b9c079dc5c98eb73d66fc7ef70baf18ad2dc3141dbb8c6a9f"} Oct 30 19:25:10.645298 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:10.645263 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-75znd" event={"ID":"7146ed33-b8b4-4a00-806a-3fec768db24a","Type":"ContainerStarted","Data":"57d4224e8c972714597b3e2137ea1a054c3fce8facf32916e387e89074a7a415"} Oct 30 19:25:10.646496 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:10.646475 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c7579b978-jsqnr" event={"ID":"a94b1abe-f37a-439a-98e1-8e82038078ff","Type":"ContainerStarted","Data":"e46a2d97c071fa0f7c1447c2b463899253d9a40e9b8b7ca48f265a0fbca63d26"} Oct 30 19:25:10.647705 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:10.647686 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-flg7z" event={"ID":"d7e50900-a49f-4316-9d27-f58cd07fda3e","Type":"ContainerStarted","Data":"1dabde73e3ad88511c47ac8c1db238f5fe6b65711ddc91ef58487a8a5bed71eb"} Oct 30 19:25:10.648769 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:10.648749 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v7wb4" event={"ID":"e6cc8ab5-bbd4-4ad6-ad02-97b7efca7588","Type":"ContainerStarted","Data":"e807dc3bc38251c7ac4d8c04813265c5066a2f9db9e3d697e0769e7f48203ff4"} Oct 30 19:25:10.650171 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:10.650153 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kwj6c" event={"ID":"3b1dddd9-2a1f-4a8a-bf37-a1e59b2a0834","Type":"ContainerStarted","Data":"786570f773b62ee6af1c451873d1074f0e07d134168da449c50010fb28ceaed5"} Oct 30 19:25:10.650265 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:10.650201 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kwj6c" event={"ID":"3b1dddd9-2a1f-4a8a-bf37-a1e59b2a0834","Type":"ContainerStarted","Data":"01a4cdfb3f644d1caad202692cd88bc0e3c2a502265a764c69d89414e0ed0f31"} Oct 30 19:25:10.758584 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:10.758484 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-76bf57bff9-zwlxg" podStartSLOduration=1.758468927 podStartE2EDuration="1.758468927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:10.758168049 +0000 UTC m=+14.276483518" watchObservedRunningTime="2026-10-30 19:25:10.758468927 +0000 UTC m=+14.276784415" Oct 30 19:25:11.658457 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:11.658417 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kwj6c" event={"ID":"3b1dddd9-2a1f-4a8a-bf37-a1e59b2a0834","Type":"ContainerStarted","Data":"ebd266bca251c8c70758c1b6b559ed19a8bb80d3d33ffe82aa1ff07124786f5a"} Oct 30 19:25:12.665480 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:12.665440 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-flg7z" event={"ID":"d7e50900-a49f-4316-9d27-f58cd07fda3e","Type":"ContainerStarted","Data":"8f35618402c9eb66fbf46377744be607c5b2a1b06cdf37f8a42379a66f7d1b90"} Oct 30 19:25:12.666515 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:12.666492 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-flg7z" Oct 30 19:25:12.676100 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:12.676075 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-flg7z" Oct 30 19:25:12.776897 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:12.776722 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-flg7z" podStartSLOduration=1.873792001 podStartE2EDuration="3.776702412s" totalImagesPullingTime="1.902910411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:12.775832238 +0000 UTC m=+16.294147696" watchObservedRunningTime="2026-10-30 19:25:12.776702412 +0000 UTC m=+16.295017870" Oct 30 19:25:13.677075 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:13.676928 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v7wb4" event={"ID":"e6cc8ab5-bbd4-4ad6-ad02-97b7efca7588","Type":"ContainerStarted","Data":"c0ed1533f91b88f1c5c3c903d88bc6a326d8d53e693885fcd597ed81f56812c2"} Oct 30 19:25:13.677075 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:13.676981 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v7wb4" event={"ID":"e6cc8ab5-bbd4-4ad6-ad02-97b7efca7588","Type":"ContainerStarted","Data":"f04c7a1344120b5d794fb48b7a51cf169bc5d8ca1c98d7cfbdad0ac1f20ac9ca"} Oct 30 19:25:13.677075 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:13.677029 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-v7wb4" Oct 30 19:25:14.414087 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:14.413826 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-v7wb4" podStartSLOduration=2.378674458 podStartE2EDuration="5.413805468s" totalImagesPullingTime="3.03513101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:13.795520647 +0000 UTC m=+17.313836121" watchObservedRunningTime="2026-10-30 19:25:14.413805468 +0000 UTC m=+17.932120923" Oct 30 19:25:14.414930 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:14.414904 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-7pqbj"] Oct 30 19:25:14.418535 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:14.418514 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-7pqbj" Oct 30 19:25:14.469042 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:14.469014 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 30 19:25:14.506676 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:14.506626 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45z9t\" (UniqueName: \"kubernetes.io/projected/a394616c-7264-42c7-9774-0e38966b6b14-kube-api-access-45z9t\") pod \"prometheus-operator-9cd89b74c-7pqbj\" (UID: \"a394616c-7264-42c7-9774-0e38966b6b14\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7pqbj" Oct 30 19:25:14.506676 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:14.506681 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a394616c-7264-42c7-9774-0e38966b6b14-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-7pqbj\" (UID: \"a394616c-7264-42c7-9774-0e38966b6b14\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7pqbj" Oct 30 19:25:14.506907 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:14.506708 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a394616c-7264-42c7-9774-0e38966b6b14-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-7pqbj\" (UID: \"a394616c-7264-42c7-9774-0e38966b6b14\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7pqbj" Oct 30 19:25:14.506907 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:14.506752 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a394616c-7264-42c7-9774-0e38966b6b14-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-7pqbj\" (UID: \"a394616c-7264-42c7-9774-0e38966b6b14\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7pqbj" Oct 30 19:25:14.514922 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:14.514891 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-8l45g\"" Oct 30 19:25:14.515091 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:14.514997 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 19:25:14.515091 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:14.515054 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 19:25:14.515091 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:14.515068 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 30 19:25:14.515244 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:14.515202 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 19:25:14.516686 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:14.516599 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-7pqbj"] Oct 30 19:25:14.608157 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:14.608119 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a394616c-7264-42c7-9774-0e38966b6b14-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-7pqbj\" (UID: \"a394616c-7264-42c7-9774-0e38966b6b14\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7pqbj" Oct 30 19:25:14.608344 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:14.608204 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45z9t\" (UniqueName: \"kubernetes.io/projected/a394616c-7264-42c7-9774-0e38966b6b14-kube-api-access-45z9t\") pod \"prometheus-operator-9cd89b74c-7pqbj\" (UID: \"a394616c-7264-42c7-9774-0e38966b6b14\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7pqbj" Oct 30 19:25:14.608344 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:14.608229 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a394616c-7264-42c7-9774-0e38966b6b14-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-7pqbj\" (UID: \"a394616c-7264-42c7-9774-0e38966b6b14\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7pqbj" Oct 30 19:25:14.608344 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:14.608266 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a394616c-7264-42c7-9774-0e38966b6b14-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-7pqbj\" (UID: \"a394616c-7264-42c7-9774-0e38966b6b14\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7pqbj" Oct 30 19:25:14.610150 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:14.610124 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a394616c-7264-42c7-9774-0e38966b6b14-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-7pqbj\" (UID: \"a394616c-7264-42c7-9774-0e38966b6b14\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7pqbj" Oct 30 19:25:14.612312 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:14.612283 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a394616c-7264-42c7-9774-0e38966b6b14-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-7pqbj\" (UID: \"a394616c-7264-42c7-9774-0e38966b6b14\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7pqbj" Oct 30 19:25:14.613891 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:14.613864 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a394616c-7264-42c7-9774-0e38966b6b14-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-7pqbj\" (UID: \"a394616c-7264-42c7-9774-0e38966b6b14\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7pqbj" Oct 30 19:25:14.675684 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:14.675607 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45z9t\" (UniqueName: \"kubernetes.io/projected/a394616c-7264-42c7-9774-0e38966b6b14-kube-api-access-45z9t\") pod \"prometheus-operator-9cd89b74c-7pqbj\" (UID: \"a394616c-7264-42c7-9774-0e38966b6b14\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7pqbj" Oct 30 19:25:14.690450 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:14.690413 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kwj6c" event={"ID":"3b1dddd9-2a1f-4a8a-bf37-a1e59b2a0834","Type":"ContainerStarted","Data":"4534f75cb6bb6093465bae79fcf727d01393960c5048e9e6d0befad6bd04b274"} Oct 30 19:25:14.733056 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:14.733031 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-7pqbj" Oct 30 19:25:14.805823 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:14.805770 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-kwj6c" podStartSLOduration=2.026263872 podStartE2EDuration="5.805749278s" totalImagesPullingTime="3.779485406s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:25:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:14.803077702 +0000 UTC m=+18.321393182" watchObservedRunningTime="2026-10-30 19:25:14.805749278 +0000 UTC m=+18.324064737" Oct 30 19:25:15.120520 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:15.120481 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-7pqbj"] Oct 30 19:25:15.128689 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:25:15.128655 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda394616c_7264_42c7_9774_0e38966b6b14.slice/crio-91bf8de61a017ac8a872cf92030775cd80ed4ef4a6746598a6f453a50ec93124 WatchSource:0}: Error finding container 91bf8de61a017ac8a872cf92030775cd80ed4ef4a6746598a6f453a50ec93124: Status 404 returned error can't find the container with id 91bf8de61a017ac8a872cf92030775cd80ed4ef4a6746598a6f453a50ec93124 Oct 30 19:25:15.695652 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:15.695613 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-75znd" event={"ID":"7146ed33-b8b4-4a00-806a-3fec768db24a","Type":"ContainerStarted","Data":"6d7cfb6d667d39324156f2b8a21832649c02fd9a714d62cb80bbe6cdf587843d"} Oct 30 19:25:15.697399 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:15.697366 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c7579b978-jsqnr" event={"ID":"a94b1abe-f37a-439a-98e1-8e82038078ff","Type":"ContainerStarted","Data":"8b317b0986b8a9c7b1d477f7704af1a18ef821b08e4eaf12b15eaeedd548b526"} Oct 30 19:25:15.698717 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:15.698691 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-7pqbj" event={"ID":"a394616c-7264-42c7-9774-0e38966b6b14","Type":"ContainerStarted","Data":"91bf8de61a017ac8a872cf92030775cd80ed4ef4a6746598a6f453a50ec93124"} Oct 30 19:25:15.922828 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:15.922773 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6c7579b978-jsqnr" podStartSLOduration=1.922148551 podStartE2EDuration="6.92275658s" totalImagesPullingTime="5.000608029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:15.92212732 +0000 UTC m=+19.440442800" watchObservedRunningTime="2026-10-30 19:25:15.92275658 +0000 UTC m=+19.441072037" Oct 30 19:25:15.923067 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:15.923045 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-75znd" podStartSLOduration=2.415282843 podStartE2EDuration="6.923039173s" totalImagesPullingTime="4.50775633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:15.806647334 +0000 UTC m=+19.324962818" watchObservedRunningTime="2026-10-30 19:25:15.923039173 +0000 UTC m=+19.441354630" Oct 30 19:25:17.709017 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:17.708973 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-7pqbj" event={"ID":"a394616c-7264-42c7-9774-0e38966b6b14","Type":"ContainerStarted","Data":"d061f326c6152f510e25272dec7eb899c7fa3c8b46b48f129375c7c032dfa227"} Oct 30 19:25:17.709498 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:17.709074 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-7pqbj" event={"ID":"a394616c-7264-42c7-9774-0e38966b6b14","Type":"ContainerStarted","Data":"11b251efc3506fb866220a5865eca27486abd4a576bc249c85f4fb8478d67a63"} Oct 30 19:25:17.834560 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:17.833726 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-7pqbj" podStartSLOduration=2.198641068 podStartE2EDuration="3.833706806s" totalImagesPullingTime="1.635065738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:17.833180142 +0000 UTC m=+21.351495633" watchObservedRunningTime="2026-10-30 19:25:17.833706806 +0000 UTC m=+21.352022264" Oct 30 19:25:19.721818 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:19.721782 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6c7579b978-jsqnr" Oct 30 19:25:19.722286 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:19.721844 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c7579b978-jsqnr" Oct 30 19:25:19.727643 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:19.727614 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6c7579b978-jsqnr" Oct 30 19:25:20.724145 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:20.724117 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6c7579b978-jsqnr" Oct 30 19:25:21.193945 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:21.193911 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2d8dk"] Oct 30 19:25:21.196374 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:21.196330 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2d8dk" Oct 30 19:25:21.251027 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:21.250996 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 19:25:21.266654 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:21.266612 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/972b92b5-ad64-48c7-b124-28d535b4b6ac-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2d8dk\" (UID: \"972b92b5-ad64-48c7-b124-28d535b4b6ac\") " pod="openshift-monitoring/node-exporter-2d8dk" Oct 30 19:25:21.266832 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:21.266666 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/972b92b5-ad64-48c7-b124-28d535b4b6ac-metrics-client-ca\") pod \"node-exporter-2d8dk\" (UID: \"972b92b5-ad64-48c7-b124-28d535b4b6ac\") " pod="openshift-monitoring/node-exporter-2d8dk" Oct 30 19:25:21.266832 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:21.266753 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/972b92b5-ad64-48c7-b124-28d535b4b6ac-root\") pod \"node-exporter-2d8dk\" (UID: \"972b92b5-ad64-48c7-b124-28d535b4b6ac\") " pod="openshift-monitoring/node-exporter-2d8dk" Oct 30 19:25:21.266832 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:21.266785 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/972b92b5-ad64-48c7-b124-28d535b4b6ac-node-exporter-wtmp\") pod \"node-exporter-2d8dk\" (UID: \"972b92b5-ad64-48c7-b124-28d535b4b6ac\") " pod="openshift-monitoring/node-exporter-2d8dk" Oct 30 19:25:21.266832 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:21.266820 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/972b92b5-ad64-48c7-b124-28d535b4b6ac-node-exporter-tls\") pod \"node-exporter-2d8dk\" (UID: \"972b92b5-ad64-48c7-b124-28d535b4b6ac\") " pod="openshift-monitoring/node-exporter-2d8dk" Oct 30 19:25:21.267053 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:21.266934 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/972b92b5-ad64-48c7-b124-28d535b4b6ac-node-exporter-accelerators-collector-config\") pod \"node-exporter-2d8dk\" (UID: \"972b92b5-ad64-48c7-b124-28d535b4b6ac\") " pod="openshift-monitoring/node-exporter-2d8dk" Oct 30 19:25:21.267053 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:21.266964 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/972b92b5-ad64-48c7-b124-28d535b4b6ac-sys\") pod \"node-exporter-2d8dk\" (UID: \"972b92b5-ad64-48c7-b124-28d535b4b6ac\") " pod="openshift-monitoring/node-exporter-2d8dk" Oct 30 19:25:21.267053 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:21.266995 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/972b92b5-ad64-48c7-b124-28d535b4b6ac-node-exporter-textfile\") pod \"node-exporter-2d8dk\" (UID: \"972b92b5-ad64-48c7-b124-28d535b4b6ac\") " pod="openshift-monitoring/node-exporter-2d8dk" Oct 30 19:25:21.267053 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:21.267014 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d2bw\" (UniqueName: \"kubernetes.io/projected/972b92b5-ad64-48c7-b124-28d535b4b6ac-kube-api-access-5d2bw\") pod \"node-exporter-2d8dk\" (UID: \"972b92b5-ad64-48c7-b124-28d535b4b6ac\") " pod="openshift-monitoring/node-exporter-2d8dk" Oct 30 19:25:21.297005 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:21.296975 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-8mnv5\"" Oct 30 19:25:21.297005 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:21.296977 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 19:25:21.343767 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:21.343725 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 19:25:21.368080 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:21.368040 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/972b92b5-ad64-48c7-b124-28d535b4b6ac-sys\") pod \"node-exporter-2d8dk\" (UID: \"972b92b5-ad64-48c7-b124-28d535b4b6ac\") " pod="openshift-monitoring/node-exporter-2d8dk" Oct 30 19:25:21.368080 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:21.368085 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/972b92b5-ad64-48c7-b124-28d535b4b6ac-node-exporter-textfile\") pod \"node-exporter-2d8dk\" (UID: \"972b92b5-ad64-48c7-b124-28d535b4b6ac\") " pod="openshift-monitoring/node-exporter-2d8dk" Oct 30 19:25:21.368307 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:21.368112 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d2bw\" (UniqueName: \"kubernetes.io/projected/972b92b5-ad64-48c7-b124-28d535b4b6ac-kube-api-access-5d2bw\") pod \"node-exporter-2d8dk\" (UID: \"972b92b5-ad64-48c7-b124-28d535b4b6ac\") " pod="openshift-monitoring/node-exporter-2d8dk" Oct 30 19:25:21.368307 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:21.368140 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/972b92b5-ad64-48c7-b124-28d535b4b6ac-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2d8dk\" (UID: \"972b92b5-ad64-48c7-b124-28d535b4b6ac\") " pod="openshift-monitoring/node-exporter-2d8dk" Oct 30 19:25:21.368307 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:21.368155 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/972b92b5-ad64-48c7-b124-28d535b4b6ac-sys\") pod \"node-exporter-2d8dk\" (UID: \"972b92b5-ad64-48c7-b124-28d535b4b6ac\") " pod="openshift-monitoring/node-exporter-2d8dk" Oct 30 19:25:21.368307 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:21.368171 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/972b92b5-ad64-48c7-b124-28d535b4b6ac-metrics-client-ca\") pod \"node-exporter-2d8dk\" (UID: \"972b92b5-ad64-48c7-b124-28d535b4b6ac\") " pod="openshift-monitoring/node-exporter-2d8dk" Oct 30 19:25:21.368307 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:21.368198 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/972b92b5-ad64-48c7-b124-28d535b4b6ac-root\") pod \"node-exporter-2d8dk\" (UID: \"972b92b5-ad64-48c7-b124-28d535b4b6ac\") " pod="openshift-monitoring/node-exporter-2d8dk" Oct 30 19:25:21.368307 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:21.368246 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/972b92b5-ad64-48c7-b124-28d535b4b6ac-root\") pod \"node-exporter-2d8dk\" (UID: \"972b92b5-ad64-48c7-b124-28d535b4b6ac\") " pod="openshift-monitoring/node-exporter-2d8dk" Oct 30 19:25:21.368307 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:21.368267 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/972b92b5-ad64-48c7-b124-28d535b4b6ac-node-exporter-wtmp\") pod \"node-exporter-2d8dk\" (UID: \"972b92b5-ad64-48c7-b124-28d535b4b6ac\") " pod="openshift-monitoring/node-exporter-2d8dk" Oct 30 19:25:21.368674 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:21.368311 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/972b92b5-ad64-48c7-b124-28d535b4b6ac-node-exporter-tls\") pod \"node-exporter-2d8dk\" (UID: \"972b92b5-ad64-48c7-b124-28d535b4b6ac\") " pod="openshift-monitoring/node-exporter-2d8dk" Oct 30 19:25:21.368674 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:21.368373 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/972b92b5-ad64-48c7-b124-28d535b4b6ac-node-exporter-accelerators-collector-config\") pod \"node-exporter-2d8dk\" (UID: \"972b92b5-ad64-48c7-b124-28d535b4b6ac\") " pod="openshift-monitoring/node-exporter-2d8dk" Oct 30 19:25:21.368674 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:21.368489 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/972b92b5-ad64-48c7-b124-28d535b4b6ac-node-exporter-wtmp\") pod \"node-exporter-2d8dk\" (UID: \"972b92b5-ad64-48c7-b124-28d535b4b6ac\") " pod="openshift-monitoring/node-exporter-2d8dk" Oct 30 19:25:21.368674 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:21.368550 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/972b92b5-ad64-48c7-b124-28d535b4b6ac-node-exporter-textfile\") pod \"node-exporter-2d8dk\" (UID: \"972b92b5-ad64-48c7-b124-28d535b4b6ac\") " pod="openshift-monitoring/node-exporter-2d8dk" Oct 30 19:25:21.368870 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:21.368853 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/972b92b5-ad64-48c7-b124-28d535b4b6ac-metrics-client-ca\") pod \"node-exporter-2d8dk\" (UID: \"972b92b5-ad64-48c7-b124-28d535b4b6ac\") " pod="openshift-monitoring/node-exporter-2d8dk" Oct 30 19:25:21.369317 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:21.369266 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/972b92b5-ad64-48c7-b124-28d535b4b6ac-node-exporter-accelerators-collector-config\") pod \"node-exporter-2d8dk\" (UID: \"972b92b5-ad64-48c7-b124-28d535b4b6ac\") " pod="openshift-monitoring/node-exporter-2d8dk" Oct 30 19:25:21.370979 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:21.370956 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/972b92b5-ad64-48c7-b124-28d535b4b6ac-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2d8dk\" (UID: \"972b92b5-ad64-48c7-b124-28d535b4b6ac\") " pod="openshift-monitoring/node-exporter-2d8dk" Oct 30 19:25:21.371075 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:21.370962 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/972b92b5-ad64-48c7-b124-28d535b4b6ac-node-exporter-tls\") pod \"node-exporter-2d8dk\" (UID: \"972b92b5-ad64-48c7-b124-28d535b4b6ac\") " pod="openshift-monitoring/node-exporter-2d8dk" Oct 30 19:25:21.429664 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:21.429627 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d2bw\" (UniqueName: \"kubernetes.io/projected/972b92b5-ad64-48c7-b124-28d535b4b6ac-kube-api-access-5d2bw\") pod \"node-exporter-2d8dk\" (UID: \"972b92b5-ad64-48c7-b124-28d535b4b6ac\") " pod="openshift-monitoring/node-exporter-2d8dk" Oct 30 19:25:21.510762 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:21.510676 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2d8dk" Oct 30 19:25:21.521209 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:25:21.521176 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod972b92b5_ad64_48c7_b124_28d535b4b6ac.slice/crio-5c55d2bd67d2831ea459295f7ff2d3780d5364c9807fce1064c66fd79e5a5771 WatchSource:0}: Error finding container 5c55d2bd67d2831ea459295f7ff2d3780d5364c9807fce1064c66fd79e5a5771: Status 404 returned error can't find the container with id 5c55d2bd67d2831ea459295f7ff2d3780d5364c9807fce1064c66fd79e5a5771 Oct 30 19:25:21.728032 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:21.727986 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2d8dk" event={"ID":"972b92b5-ad64-48c7-b124-28d535b4b6ac","Type":"ContainerStarted","Data":"5c55d2bd67d2831ea459295f7ff2d3780d5364c9807fce1064c66fd79e5a5771"} Oct 30 19:25:21.799162 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:21.799133 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-l66t6" Oct 30 19:25:23.737651 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:23.737613 4292 generic.go:356] "Generic (PLEG): container finished" podID="972b92b5-ad64-48c7-b124-28d535b4b6ac" containerID="2f51eeff48f8f2674b60a93194f93050a10d429bf90c5d3a314b7a885be73f55" exitCode=0 Oct 30 19:25:23.738187 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:23.737712 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2d8dk" event={"ID":"972b92b5-ad64-48c7-b124-28d535b4b6ac","Type":"ContainerDied","Data":"2f51eeff48f8f2674b60a93194f93050a10d429bf90c5d3a314b7a885be73f55"} Oct 30 19:25:24.701750 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:24.701718 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-v7wb4" Oct 30 19:25:24.744299 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:24.744262 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2d8dk" event={"ID":"972b92b5-ad64-48c7-b124-28d535b4b6ac","Type":"ContainerStarted","Data":"c49036c5b37e678fe8d8102926b515aefa61f08e1d49f90bcd7c83b0cb812d09"} Oct 30 19:25:24.744747 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:24.744326 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2d8dk" event={"ID":"972b92b5-ad64-48c7-b124-28d535b4b6ac","Type":"ContainerStarted","Data":"96aab150001179f931513c8efe1dec74dd05b8edd3c332b26211f403af9fba52"} Oct 30 19:25:24.934212 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:24.934149 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2d8dk" podStartSLOduration=2.4720056 podStartE2EDuration="3.934131485s" totalImagesPullingTime="1.384800343s" totalInitContainerRuntime="77.325542ms" isStatefulPod=true podCreationTimestamp="2026-10-30 19:25:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:24.933343368 +0000 UTC m=+28.451658825" watchObservedRunningTime="2026-10-30 19:25:24.934131485 +0000 UTC m=+28.452446941" Oct 30 19:25:26.081793 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:26.081757 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-t6m9p" Oct 30 19:25:27.986195 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:27.986164 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c7579b978-jsqnr"] Oct 30 19:25:28.763453 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:28.763417 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-zg48z" event={"ID":"28caab70-818c-447e-8974-fd3d5ef4b762","Type":"ContainerStarted","Data":"e60c0a75e5904e425fac9e11bde1868eb5efb7d8790a2495ed157af5a592e084"} Oct 30 19:25:28.764378 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:28.764336 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-zg48z" Oct 30 19:25:28.774414 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:28.774390 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-zg48z" Oct 30 19:25:28.876671 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:28.876617 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-zg48z" podStartSLOduration=1.797404228 podStartE2EDuration="19.876598121s" totalImagesPullingTime="18.079193893s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:28.875810358 +0000 UTC m=+32.394125829" watchObservedRunningTime="2026-10-30 19:25:28.876598121 +0000 UTC m=+32.394913582" Oct 30 19:25:31.663773 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:31.663743 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-76bf57bff9-zwlxg" Oct 30 19:25:31.685573 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:31.685538 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5c976d4497-497d9"] Oct 30 19:25:31.710759 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:31.710726 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c976d4497-497d9" Oct 30 19:25:31.762850 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:31.762818 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c976d4497-497d9"] Oct 30 19:25:31.860014 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:31.859977 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7bcd7c61-a1be-4682-8168-f59051e470ec-trusted-ca-bundle\") pod \"console-5c976d4497-497d9\" (UID: \"7bcd7c61-a1be-4682-8168-f59051e470ec\") " pod="openshift-console/console-5c976d4497-497d9" Oct 30 19:25:31.860191 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:31.860022 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7bcd7c61-a1be-4682-8168-f59051e470ec-oauth-serving-cert\") pod \"console-5c976d4497-497d9\" (UID: \"7bcd7c61-a1be-4682-8168-f59051e470ec\") " pod="openshift-console/console-5c976d4497-497d9" Oct 30 19:25:31.860191 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:31.860125 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7bcd7c61-a1be-4682-8168-f59051e470ec-console-config\") pod \"console-5c976d4497-497d9\" (UID: \"7bcd7c61-a1be-4682-8168-f59051e470ec\") " pod="openshift-console/console-5c976d4497-497d9" Oct 30 19:25:31.860191 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:31.860154 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7bcd7c61-a1be-4682-8168-f59051e470ec-service-ca\") pod \"console-5c976d4497-497d9\" (UID: \"7bcd7c61-a1be-4682-8168-f59051e470ec\") " pod="openshift-console/console-5c976d4497-497d9" Oct 30 19:25:31.860191 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:31.860183 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7bcd7c61-a1be-4682-8168-f59051e470ec-console-serving-cert\") pod \"console-5c976d4497-497d9\" (UID: \"7bcd7c61-a1be-4682-8168-f59051e470ec\") " pod="openshift-console/console-5c976d4497-497d9" Oct 30 19:25:31.860455 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:31.860219 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7bcd7c61-a1be-4682-8168-f59051e470ec-tmp\") pod \"console-5c976d4497-497d9\" (UID: \"7bcd7c61-a1be-4682-8168-f59051e470ec\") " pod="openshift-console/console-5c976d4497-497d9" Oct 30 19:25:31.860455 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:31.860246 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krkmh\" (UniqueName: \"kubernetes.io/projected/7bcd7c61-a1be-4682-8168-f59051e470ec-kube-api-access-krkmh\") pod \"console-5c976d4497-497d9\" (UID: \"7bcd7c61-a1be-4682-8168-f59051e470ec\") " pod="openshift-console/console-5c976d4497-497d9" Oct 30 19:25:31.860455 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:31.860299 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7bcd7c61-a1be-4682-8168-f59051e470ec-console-oauth-config\") pod \"console-5c976d4497-497d9\" (UID: \"7bcd7c61-a1be-4682-8168-f59051e470ec\") " pod="openshift-console/console-5c976d4497-497d9" Oct 30 19:25:31.872233 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:31.872204 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 19:25:31.961716 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:31.961630 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7bcd7c61-a1be-4682-8168-f59051e470ec-console-serving-cert\") pod \"console-5c976d4497-497d9\" (UID: \"7bcd7c61-a1be-4682-8168-f59051e470ec\") " pod="openshift-console/console-5c976d4497-497d9" Oct 30 19:25:31.961716 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:31.961680 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7bcd7c61-a1be-4682-8168-f59051e470ec-tmp\") pod \"console-5c976d4497-497d9\" (UID: \"7bcd7c61-a1be-4682-8168-f59051e470ec\") " pod="openshift-console/console-5c976d4497-497d9" Oct 30 19:25:31.961716 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:31.961707 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krkmh\" (UniqueName: \"kubernetes.io/projected/7bcd7c61-a1be-4682-8168-f59051e470ec-kube-api-access-krkmh\") pod \"console-5c976d4497-497d9\" (UID: \"7bcd7c61-a1be-4682-8168-f59051e470ec\") " pod="openshift-console/console-5c976d4497-497d9" Oct 30 19:25:31.961977 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:31.961757 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7bcd7c61-a1be-4682-8168-f59051e470ec-console-oauth-config\") pod \"console-5c976d4497-497d9\" (UID: \"7bcd7c61-a1be-4682-8168-f59051e470ec\") " pod="openshift-console/console-5c976d4497-497d9" Oct 30 19:25:31.961977 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:31.961813 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7bcd7c61-a1be-4682-8168-f59051e470ec-trusted-ca-bundle\") pod \"console-5c976d4497-497d9\" (UID: \"7bcd7c61-a1be-4682-8168-f59051e470ec\") " pod="openshift-console/console-5c976d4497-497d9" Oct 30 19:25:31.961977 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:31.961834 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7bcd7c61-a1be-4682-8168-f59051e470ec-oauth-serving-cert\") pod \"console-5c976d4497-497d9\" (UID: \"7bcd7c61-a1be-4682-8168-f59051e470ec\") " pod="openshift-console/console-5c976d4497-497d9" Oct 30 19:25:31.961977 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:31.961861 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7bcd7c61-a1be-4682-8168-f59051e470ec-console-config\") pod \"console-5c976d4497-497d9\" (UID: \"7bcd7c61-a1be-4682-8168-f59051e470ec\") " pod="openshift-console/console-5c976d4497-497d9" Oct 30 19:25:31.961977 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:31.961897 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7bcd7c61-a1be-4682-8168-f59051e470ec-service-ca\") pod \"console-5c976d4497-497d9\" (UID: \"7bcd7c61-a1be-4682-8168-f59051e470ec\") " pod="openshift-console/console-5c976d4497-497d9" Oct 30 19:25:31.963586 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:31.962626 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7bcd7c61-a1be-4682-8168-f59051e470ec-tmp\") pod \"console-5c976d4497-497d9\" (UID: \"7bcd7c61-a1be-4682-8168-f59051e470ec\") " pod="openshift-console/console-5c976d4497-497d9" Oct 30 19:25:31.963586 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:31.962689 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7bcd7c61-a1be-4682-8168-f59051e470ec-service-ca\") pod \"console-5c976d4497-497d9\" (UID: \"7bcd7c61-a1be-4682-8168-f59051e470ec\") " pod="openshift-console/console-5c976d4497-497d9" Oct 30 19:25:31.963586 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:31.962953 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7bcd7c61-a1be-4682-8168-f59051e470ec-oauth-serving-cert\") pod \"console-5c976d4497-497d9\" (UID: \"7bcd7c61-a1be-4682-8168-f59051e470ec\") " pod="openshift-console/console-5c976d4497-497d9" Oct 30 19:25:31.963586 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:31.963404 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7bcd7c61-a1be-4682-8168-f59051e470ec-console-config\") pod \"console-5c976d4497-497d9\" (UID: \"7bcd7c61-a1be-4682-8168-f59051e470ec\") " pod="openshift-console/console-5c976d4497-497d9" Oct 30 19:25:31.963586 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:31.963544 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7bcd7c61-a1be-4682-8168-f59051e470ec-trusted-ca-bundle\") pod \"console-5c976d4497-497d9\" (UID: \"7bcd7c61-a1be-4682-8168-f59051e470ec\") " pod="openshift-console/console-5c976d4497-497d9" Oct 30 19:25:31.965305 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:31.965282 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7bcd7c61-a1be-4682-8168-f59051e470ec-console-oauth-config\") pod \"console-5c976d4497-497d9\" (UID: \"7bcd7c61-a1be-4682-8168-f59051e470ec\") " pod="openshift-console/console-5c976d4497-497d9" Oct 30 19:25:31.965696 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:31.965679 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7bcd7c61-a1be-4682-8168-f59051e470ec-console-serving-cert\") pod \"console-5c976d4497-497d9\" (UID: \"7bcd7c61-a1be-4682-8168-f59051e470ec\") " pod="openshift-console/console-5c976d4497-497d9" Oct 30 19:25:32.031137 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:32.031104 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krkmh\" (UniqueName: \"kubernetes.io/projected/7bcd7c61-a1be-4682-8168-f59051e470ec-kube-api-access-krkmh\") pod \"console-5c976d4497-497d9\" (UID: \"7bcd7c61-a1be-4682-8168-f59051e470ec\") " pod="openshift-console/console-5c976d4497-497d9" Oct 30 19:25:32.041004 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:32.040976 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c976d4497-497d9" Oct 30 19:25:32.447300 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:32.447257 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c976d4497-497d9"] Oct 30 19:25:32.454491 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:25:32.454458 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bcd7c61_a1be_4682_8168_f59051e470ec.slice/crio-ba084e4b99c32f681d58054896d5ba5fdec8c05fb5c81f750e04f4125e96d921 WatchSource:0}: Error finding container ba084e4b99c32f681d58054896d5ba5fdec8c05fb5c81f750e04f4125e96d921: Status 404 returned error can't find the container with id ba084e4b99c32f681d58054896d5ba5fdec8c05fb5c81f750e04f4125e96d921 Oct 30 19:25:32.779796 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:32.779754 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c976d4497-497d9" event={"ID":"7bcd7c61-a1be-4682-8168-f59051e470ec","Type":"ContainerStarted","Data":"c79b6130cf60311c65eb3108b8ddccd9e3fa0117786dc2373340962a4fc5d43f"} Oct 30 19:25:32.780269 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:32.779820 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c976d4497-497d9" event={"ID":"7bcd7c61-a1be-4682-8168-f59051e470ec","Type":"ContainerStarted","Data":"ba084e4b99c32f681d58054896d5ba5fdec8c05fb5c81f750e04f4125e96d921"} Oct 30 19:25:32.893787 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:32.893725 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5c976d4497-497d9" podStartSLOduration=1.893708865 podStartE2EDuration="1.893708865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:32.89283608 +0000 UTC m=+36.411151538" watchObservedRunningTime="2026-10-30 19:25:32.893708865 +0000 UTC m=+36.412024321" Oct 30 19:25:38.151961 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:38.151917 4292 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l66t6_c9eecef1-2e58-4444-8a51-25b0b18715f5/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:38.376723 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:38.376691 4292 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l66t6_c9eecef1-2e58-4444-8a51-25b0b18715f5/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:38.615337 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:38.615303 4292 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l66t6_c9eecef1-2e58-4444-8a51-25b0b18715f5/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:42.042515 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:42.042416 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5c976d4497-497d9" Oct 30 19:25:42.042515 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:42.042492 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5c976d4497-497d9" Oct 30 19:25:42.047523 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:42.047498 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5c976d4497-497d9" Oct 30 19:25:42.820291 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:42.820264 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5c976d4497-497d9" Oct 30 19:25:43.219591 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:25:43.219517 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:25:43.390223 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:43.390188 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-57899bb8b4-4tvlj"] Oct 30 19:25:43.419874 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:43.419846 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57899bb8b4-4tvlj" Oct 30 19:25:43.452837 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:43.452809 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f0f9c9ce-290b-4c9c-8ffc-fd7cbcca0615-oauth-serving-cert\") pod \"console-57899bb8b4-4tvlj\" (UID: \"f0f9c9ce-290b-4c9c-8ffc-fd7cbcca0615\") " pod="openshift-console/console-57899bb8b4-4tvlj" Oct 30 19:25:43.452951 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:43.452841 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f0f9c9ce-290b-4c9c-8ffc-fd7cbcca0615-console-oauth-config\") pod \"console-57899bb8b4-4tvlj\" (UID: \"f0f9c9ce-290b-4c9c-8ffc-fd7cbcca0615\") " pod="openshift-console/console-57899bb8b4-4tvlj" Oct 30 19:25:43.452951 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:43.452910 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f0f9c9ce-290b-4c9c-8ffc-fd7cbcca0615-tmp\") pod \"console-57899bb8b4-4tvlj\" (UID: \"f0f9c9ce-290b-4c9c-8ffc-fd7cbcca0615\") " pod="openshift-console/console-57899bb8b4-4tvlj" Oct 30 19:25:43.452951 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:43.452939 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzdgl\" (UniqueName: \"kubernetes.io/projected/f0f9c9ce-290b-4c9c-8ffc-fd7cbcca0615-kube-api-access-nzdgl\") pod \"console-57899bb8b4-4tvlj\" (UID: \"f0f9c9ce-290b-4c9c-8ffc-fd7cbcca0615\") " pod="openshift-console/console-57899bb8b4-4tvlj" Oct 30 19:25:43.453057 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:43.452960 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f0f9c9ce-290b-4c9c-8ffc-fd7cbcca0615-console-serving-cert\") pod \"console-57899bb8b4-4tvlj\" (UID: \"f0f9c9ce-290b-4c9c-8ffc-fd7cbcca0615\") " pod="openshift-console/console-57899bb8b4-4tvlj" Oct 30 19:25:43.453057 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:43.453010 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f0f9c9ce-290b-4c9c-8ffc-fd7cbcca0615-console-config\") pod \"console-57899bb8b4-4tvlj\" (UID: \"f0f9c9ce-290b-4c9c-8ffc-fd7cbcca0615\") " pod="openshift-console/console-57899bb8b4-4tvlj" Oct 30 19:25:43.453057 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:43.453043 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0f9c9ce-290b-4c9c-8ffc-fd7cbcca0615-trusted-ca-bundle\") pod \"console-57899bb8b4-4tvlj\" (UID: \"f0f9c9ce-290b-4c9c-8ffc-fd7cbcca0615\") " pod="openshift-console/console-57899bb8b4-4tvlj" Oct 30 19:25:43.453147 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:43.453129 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f0f9c9ce-290b-4c9c-8ffc-fd7cbcca0615-service-ca\") pod \"console-57899bb8b4-4tvlj\" (UID: \"f0f9c9ce-290b-4c9c-8ffc-fd7cbcca0615\") " pod="openshift-console/console-57899bb8b4-4tvlj" Oct 30 19:25:43.458072 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:43.458036 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57899bb8b4-4tvlj"] Oct 30 19:25:43.553992 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:43.553964 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f0f9c9ce-290b-4c9c-8ffc-fd7cbcca0615-oauth-serving-cert\") pod \"console-57899bb8b4-4tvlj\" (UID: \"f0f9c9ce-290b-4c9c-8ffc-fd7cbcca0615\") " pod="openshift-console/console-57899bb8b4-4tvlj" Oct 30 19:25:43.554187 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:43.554001 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f0f9c9ce-290b-4c9c-8ffc-fd7cbcca0615-console-oauth-config\") pod \"console-57899bb8b4-4tvlj\" (UID: \"f0f9c9ce-290b-4c9c-8ffc-fd7cbcca0615\") " pod="openshift-console/console-57899bb8b4-4tvlj" Oct 30 19:25:43.554187 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:43.554026 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f0f9c9ce-290b-4c9c-8ffc-fd7cbcca0615-tmp\") pod \"console-57899bb8b4-4tvlj\" (UID: \"f0f9c9ce-290b-4c9c-8ffc-fd7cbcca0615\") " pod="openshift-console/console-57899bb8b4-4tvlj" Oct 30 19:25:43.554187 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:43.554058 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzdgl\" (UniqueName: \"kubernetes.io/projected/f0f9c9ce-290b-4c9c-8ffc-fd7cbcca0615-kube-api-access-nzdgl\") pod \"console-57899bb8b4-4tvlj\" (UID: \"f0f9c9ce-290b-4c9c-8ffc-fd7cbcca0615\") " pod="openshift-console/console-57899bb8b4-4tvlj" Oct 30 19:25:43.554187 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:43.554082 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f0f9c9ce-290b-4c9c-8ffc-fd7cbcca0615-console-serving-cert\") pod \"console-57899bb8b4-4tvlj\" (UID: \"f0f9c9ce-290b-4c9c-8ffc-fd7cbcca0615\") " pod="openshift-console/console-57899bb8b4-4tvlj" Oct 30 19:25:43.554187 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:43.554102 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f0f9c9ce-290b-4c9c-8ffc-fd7cbcca0615-console-config\") pod \"console-57899bb8b4-4tvlj\" (UID: \"f0f9c9ce-290b-4c9c-8ffc-fd7cbcca0615\") " pod="openshift-console/console-57899bb8b4-4tvlj" Oct 30 19:25:43.554187 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:43.554130 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0f9c9ce-290b-4c9c-8ffc-fd7cbcca0615-trusted-ca-bundle\") pod \"console-57899bb8b4-4tvlj\" (UID: \"f0f9c9ce-290b-4c9c-8ffc-fd7cbcca0615\") " pod="openshift-console/console-57899bb8b4-4tvlj" Oct 30 19:25:43.554187 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:43.554177 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f0f9c9ce-290b-4c9c-8ffc-fd7cbcca0615-service-ca\") pod \"console-57899bb8b4-4tvlj\" (UID: \"f0f9c9ce-290b-4c9c-8ffc-fd7cbcca0615\") " pod="openshift-console/console-57899bb8b4-4tvlj" Oct 30 19:25:43.566101 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:43.566073 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f0f9c9ce-290b-4c9c-8ffc-fd7cbcca0615-tmp\") pod \"console-57899bb8b4-4tvlj\" (UID: \"f0f9c9ce-290b-4c9c-8ffc-fd7cbcca0615\") " pod="openshift-console/console-57899bb8b4-4tvlj" Oct 30 19:25:43.566223 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:43.566080 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f0f9c9ce-290b-4c9c-8ffc-fd7cbcca0615-oauth-serving-cert\") pod \"console-57899bb8b4-4tvlj\" (UID: \"f0f9c9ce-290b-4c9c-8ffc-fd7cbcca0615\") " pod="openshift-console/console-57899bb8b4-4tvlj" Oct 30 19:25:43.566277 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:43.566234 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f0f9c9ce-290b-4c9c-8ffc-fd7cbcca0615-console-config\") pod \"console-57899bb8b4-4tvlj\" (UID: \"f0f9c9ce-290b-4c9c-8ffc-fd7cbcca0615\") " pod="openshift-console/console-57899bb8b4-4tvlj" Oct 30 19:25:43.566314 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:43.566271 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f0f9c9ce-290b-4c9c-8ffc-fd7cbcca0615-service-ca\") pod \"console-57899bb8b4-4tvlj\" (UID: \"f0f9c9ce-290b-4c9c-8ffc-fd7cbcca0615\") " pod="openshift-console/console-57899bb8b4-4tvlj" Oct 30 19:25:43.566314 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:43.566295 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f0f9c9ce-290b-4c9c-8ffc-fd7cbcca0615-console-oauth-config\") pod \"console-57899bb8b4-4tvlj\" (UID: \"f0f9c9ce-290b-4c9c-8ffc-fd7cbcca0615\") " pod="openshift-console/console-57899bb8b4-4tvlj" Oct 30 19:25:43.566533 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:43.566516 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0f9c9ce-290b-4c9c-8ffc-fd7cbcca0615-trusted-ca-bundle\") pod \"console-57899bb8b4-4tvlj\" (UID: \"f0f9c9ce-290b-4c9c-8ffc-fd7cbcca0615\") " pod="openshift-console/console-57899bb8b4-4tvlj" Oct 30 19:25:43.566718 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:43.566686 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f0f9c9ce-290b-4c9c-8ffc-fd7cbcca0615-console-serving-cert\") pod \"console-57899bb8b4-4tvlj\" (UID: \"f0f9c9ce-290b-4c9c-8ffc-fd7cbcca0615\") " pod="openshift-console/console-57899bb8b4-4tvlj" Oct 30 19:25:43.612420 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:43.612393 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzdgl\" (UniqueName: \"kubernetes.io/projected/f0f9c9ce-290b-4c9c-8ffc-fd7cbcca0615-kube-api-access-nzdgl\") pod \"console-57899bb8b4-4tvlj\" (UID: \"f0f9c9ce-290b-4c9c-8ffc-fd7cbcca0615\") " pod="openshift-console/console-57899bb8b4-4tvlj" Oct 30 19:25:43.740616 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:43.740581 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57899bb8b4-4tvlj" Oct 30 19:25:44.146256 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:44.146180 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57899bb8b4-4tvlj"] Oct 30 19:25:44.153531 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:25:44.153505 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0f9c9ce_290b_4c9c_8ffc_fd7cbcca0615.slice/crio-f19bfcf259d8584292349fdc0a15412c01d06d7325621081f430cbcfd3d13ea5 WatchSource:0}: Error finding container f19bfcf259d8584292349fdc0a15412c01d06d7325621081f430cbcfd3d13ea5: Status 404 returned error can't find the container with id f19bfcf259d8584292349fdc0a15412c01d06d7325621081f430cbcfd3d13ea5 Oct 30 19:25:44.824731 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:44.824685 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57899bb8b4-4tvlj" event={"ID":"f0f9c9ce-290b-4c9c-8ffc-fd7cbcca0615","Type":"ContainerStarted","Data":"406bdf0923bff8af7a6770a004b1f234acb9993dd785cf218548fb9101e332a9"} Oct 30 19:25:44.825185 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:44.824747 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57899bb8b4-4tvlj" event={"ID":"f0f9c9ce-290b-4c9c-8ffc-fd7cbcca0615","Type":"ContainerStarted","Data":"f19bfcf259d8584292349fdc0a15412c01d06d7325621081f430cbcfd3d13ea5"} Oct 30 19:25:44.937692 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:44.937636 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-57899bb8b4-4tvlj" podStartSLOduration=1.9376217599999999 podStartE2EDuration="1.93762176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:44.936988561 +0000 UTC m=+48.455304039" watchObservedRunningTime="2026-10-30 19:25:44.93762176 +0000 UTC m=+48.455937216" Oct 30 19:25:53.006045 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:53.005981 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6c7579b978-jsqnr" podUID="a94b1abe-f37a-439a-98e1-8e82038078ff" containerName="console" containerID="cri-o://8b317b0986b8a9c7b1d477f7704af1a18ef821b08e4eaf12b15eaeedd548b526" gracePeriod=15 Oct 30 19:25:53.258963 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:53.258912 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c7579b978-jsqnr" Oct 30 19:25:53.317578 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:53.317550 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a94b1abe-f37a-439a-98e1-8e82038078ff-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a94b1abe-f37a-439a-98e1-8e82038078ff-oauth-serving-cert\") pod \"a94b1abe-f37a-439a-98e1-8e82038078ff\" (UID: \"a94b1abe-f37a-439a-98e1-8e82038078ff\") " Oct 30 19:25:53.317685 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:53.317592 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a94b1abe-f37a-439a-98e1-8e82038078ff-tmp\" (UniqueName: \"kubernetes.io/empty-dir/a94b1abe-f37a-439a-98e1-8e82038078ff-tmp\") pod \"a94b1abe-f37a-439a-98e1-8e82038078ff\" (UID: \"a94b1abe-f37a-439a-98e1-8e82038078ff\") " Oct 30 19:25:53.317685 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:53.317617 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a94b1abe-f37a-439a-98e1-8e82038078ff-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a94b1abe-f37a-439a-98e1-8e82038078ff-console-oauth-config\") pod \"a94b1abe-f37a-439a-98e1-8e82038078ff\" (UID: \"a94b1abe-f37a-439a-98e1-8e82038078ff\") " Oct 30 19:25:53.317685 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:53.317642 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a94b1abe-f37a-439a-98e1-8e82038078ff-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a94b1abe-f37a-439a-98e1-8e82038078ff-console-serving-cert\") pod \"a94b1abe-f37a-439a-98e1-8e82038078ff\" (UID: \"a94b1abe-f37a-439a-98e1-8e82038078ff\") " Oct 30 19:25:53.317898 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:53.317873 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a94b1abe-f37a-439a-98e1-8e82038078ff-tmp" pod "a94b1abe-f37a-439a-98e1-8e82038078ff" (UID: "a94b1abe-f37a-439a-98e1-8e82038078ff"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:25:53.318039 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:53.318015 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a94b1abe-f37a-439a-98e1-8e82038078ff-oauth-serving-cert" pod "a94b1abe-f37a-439a-98e1-8e82038078ff" (UID: "a94b1abe-f37a-439a-98e1-8e82038078ff"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:25:53.319775 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:53.319756 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a94b1abe-f37a-439a-98e1-8e82038078ff-console-oauth-config" pod "a94b1abe-f37a-439a-98e1-8e82038078ff" (UID: "a94b1abe-f37a-439a-98e1-8e82038078ff"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:25:53.319832 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:53.319807 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a94b1abe-f37a-439a-98e1-8e82038078ff-console-serving-cert" pod "a94b1abe-f37a-439a-98e1-8e82038078ff" (UID: "a94b1abe-f37a-439a-98e1-8e82038078ff"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:25:53.418074 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:53.418044 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a94b1abe-f37a-439a-98e1-8e82038078ff-service-ca\" (UniqueName: \"kubernetes.io/configmap/a94b1abe-f37a-439a-98e1-8e82038078ff-service-ca\") pod \"a94b1abe-f37a-439a-98e1-8e82038078ff\" (UID: \"a94b1abe-f37a-439a-98e1-8e82038078ff\") " Oct 30 19:25:53.418218 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:53.418095 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a94b1abe-f37a-439a-98e1-8e82038078ff-kube-api-access-fm8rq\" (UniqueName: \"kubernetes.io/projected/a94b1abe-f37a-439a-98e1-8e82038078ff-kube-api-access-fm8rq\") pod \"a94b1abe-f37a-439a-98e1-8e82038078ff\" (UID: \"a94b1abe-f37a-439a-98e1-8e82038078ff\") " Oct 30 19:25:53.418218 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:53.418118 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a94b1abe-f37a-439a-98e1-8e82038078ff-console-config\" (UniqueName: \"kubernetes.io/configmap/a94b1abe-f37a-439a-98e1-8e82038078ff-console-config\") pod \"a94b1abe-f37a-439a-98e1-8e82038078ff\" (UID: \"a94b1abe-f37a-439a-98e1-8e82038078ff\") " Oct 30 19:25:53.418323 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:53.418306 4292 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a94b1abe-f37a-439a-98e1-8e82038078ff-oauth-serving-cert\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:25:53.418389 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:53.418329 4292 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a94b1abe-f37a-439a-98e1-8e82038078ff-tmp\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:25:53.418389 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:53.418324 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a94b1abe-f37a-439a-98e1-8e82038078ff-service-ca" pod "a94b1abe-f37a-439a-98e1-8e82038078ff" (UID: "a94b1abe-f37a-439a-98e1-8e82038078ff"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:25:53.418389 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:53.418369 4292 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a94b1abe-f37a-439a-98e1-8e82038078ff-console-oauth-config\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:25:53.418389 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:53.418385 4292 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a94b1abe-f37a-439a-98e1-8e82038078ff-console-serving-cert\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:25:53.418570 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:53.418553 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a94b1abe-f37a-439a-98e1-8e82038078ff-console-config" pod "a94b1abe-f37a-439a-98e1-8e82038078ff" (UID: "a94b1abe-f37a-439a-98e1-8e82038078ff"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:25:53.420108 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:53.420089 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a94b1abe-f37a-439a-98e1-8e82038078ff-kube-api-access-fm8rq" pod "a94b1abe-f37a-439a-98e1-8e82038078ff" (UID: "a94b1abe-f37a-439a-98e1-8e82038078ff"). InnerVolumeSpecName "kube-api-access-fm8rq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:25:53.519602 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:53.519575 4292 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a94b1abe-f37a-439a-98e1-8e82038078ff-service-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:25:53.519602 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:53.519602 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm8rq\" (UniqueName: \"kubernetes.io/projected/a94b1abe-f37a-439a-98e1-8e82038078ff-kube-api-access-fm8rq\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:25:53.519775 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:53.519613 4292 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a94b1abe-f37a-439a-98e1-8e82038078ff-console-config\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:25:53.741428 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:53.741383 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-57899bb8b4-4tvlj" Oct 30 19:25:53.741599 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:53.741446 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-57899bb8b4-4tvlj" Oct 30 19:25:53.745852 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:53.745828 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-57899bb8b4-4tvlj" Oct 30 19:25:53.855075 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:53.854984 4292 generic.go:356] "Generic (PLEG): container finished" podID="a94b1abe-f37a-439a-98e1-8e82038078ff" containerID="8b317b0986b8a9c7b1d477f7704af1a18ef821b08e4eaf12b15eaeedd548b526" exitCode=0 Oct 30 19:25:53.855075 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:53.855051 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c7579b978-jsqnr" Oct 30 19:25:53.855301 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:53.855069 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c7579b978-jsqnr" event={"ID":"a94b1abe-f37a-439a-98e1-8e82038078ff","Type":"ContainerDied","Data":"8b317b0986b8a9c7b1d477f7704af1a18ef821b08e4eaf12b15eaeedd548b526"} Oct 30 19:25:53.855301 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:53.855161 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c7579b978-jsqnr" event={"ID":"a94b1abe-f37a-439a-98e1-8e82038078ff","Type":"ContainerDied","Data":"e46a2d97c071fa0f7c1447c2b463899253d9a40e9b8b7ca48f265a0fbca63d26"} Oct 30 19:25:53.855301 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:53.855189 4292 scope.go:122] "RemoveContainer" containerID="8b317b0986b8a9c7b1d477f7704af1a18ef821b08e4eaf12b15eaeedd548b526" Oct 30 19:25:53.859231 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:53.859212 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-57899bb8b4-4tvlj" Oct 30 19:25:53.863694 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:53.863647 4292 scope.go:122] "RemoveContainer" containerID="8b317b0986b8a9c7b1d477f7704af1a18ef821b08e4eaf12b15eaeedd548b526" Oct 30 19:25:53.863964 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:25:53.863945 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b317b0986b8a9c7b1d477f7704af1a18ef821b08e4eaf12b15eaeedd548b526\": container with ID starting with 8b317b0986b8a9c7b1d477f7704af1a18ef821b08e4eaf12b15eaeedd548b526 not found: ID does not exist" containerID="8b317b0986b8a9c7b1d477f7704af1a18ef821b08e4eaf12b15eaeedd548b526" Oct 30 19:25:53.864039 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:53.863973 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b317b0986b8a9c7b1d477f7704af1a18ef821b08e4eaf12b15eaeedd548b526"} err="failed to get container status \"8b317b0986b8a9c7b1d477f7704af1a18ef821b08e4eaf12b15eaeedd548b526\": rpc error: code = NotFound desc = could not find container \"8b317b0986b8a9c7b1d477f7704af1a18ef821b08e4eaf12b15eaeedd548b526\": container with ID starting with 8b317b0986b8a9c7b1d477f7704af1a18ef821b08e4eaf12b15eaeedd548b526 not found: ID does not exist" Oct 30 19:25:54.053205 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:54.053174 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c976d4497-497d9"] Oct 30 19:25:54.126314 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:54.126247 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c7579b978-jsqnr"] Oct 30 19:25:54.130000 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:54.129975 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6c7579b978-jsqnr"] Oct 30 19:25:54.579907 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:25:54.579873 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a94b1abe-f37a-439a-98e1-8e82038078ff" path="/var/lib/kubelet/pods/a94b1abe-f37a-439a-98e1-8e82038078ff/volumes" Oct 30 19:26:13.217941 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:26:13.217897 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:26:19.076641 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:26:19.076582 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5c976d4497-497d9" podUID="7bcd7c61-a1be-4682-8168-f59051e470ec" containerName="console" containerID="cri-o://c79b6130cf60311c65eb3108b8ddccd9e3fa0117786dc2373340962a4fc5d43f" gracePeriod=15 Oct 30 19:26:19.325057 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:26:19.325032 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c976d4497-497d9" Oct 30 19:26:19.400402 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:26:19.400324 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bcd7c61-a1be-4682-8168-f59051e470ec-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7bcd7c61-a1be-4682-8168-f59051e470ec-console-oauth-config\") pod \"7bcd7c61-a1be-4682-8168-f59051e470ec\" (UID: \"7bcd7c61-a1be-4682-8168-f59051e470ec\") " Oct 30 19:26:19.400402 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:26:19.400371 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bcd7c61-a1be-4682-8168-f59051e470ec-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bcd7c61-a1be-4682-8168-f59051e470ec-service-ca\") pod \"7bcd7c61-a1be-4682-8168-f59051e470ec\" (UID: \"7bcd7c61-a1be-4682-8168-f59051e470ec\") " Oct 30 19:26:19.400402 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:26:19.400399 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bcd7c61-a1be-4682-8168-f59051e470ec-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7bcd7c61-a1be-4682-8168-f59051e470ec-console-serving-cert\") pod \"7bcd7c61-a1be-4682-8168-f59051e470ec\" (UID: \"7bcd7c61-a1be-4682-8168-f59051e470ec\") " Oct 30 19:26:19.400625 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:26:19.400448 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bcd7c61-a1be-4682-8168-f59051e470ec-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7bcd7c61-a1be-4682-8168-f59051e470ec-trusted-ca-bundle\") pod \"7bcd7c61-a1be-4682-8168-f59051e470ec\" (UID: \"7bcd7c61-a1be-4682-8168-f59051e470ec\") " Oct 30 19:26:19.400625 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:26:19.400478 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bcd7c61-a1be-4682-8168-f59051e470ec-kube-api-access-krkmh\" (UniqueName: \"kubernetes.io/projected/7bcd7c61-a1be-4682-8168-f59051e470ec-kube-api-access-krkmh\") pod \"7bcd7c61-a1be-4682-8168-f59051e470ec\" (UID: \"7bcd7c61-a1be-4682-8168-f59051e470ec\") " Oct 30 19:26:19.400625 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:26:19.400500 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bcd7c61-a1be-4682-8168-f59051e470ec-console-config\" (UniqueName: \"kubernetes.io/configmap/7bcd7c61-a1be-4682-8168-f59051e470ec-console-config\") pod \"7bcd7c61-a1be-4682-8168-f59051e470ec\" (UID: \"7bcd7c61-a1be-4682-8168-f59051e470ec\") " Oct 30 19:26:19.400625 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:26:19.400518 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bcd7c61-a1be-4682-8168-f59051e470ec-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7bcd7c61-a1be-4682-8168-f59051e470ec-oauth-serving-cert\") pod \"7bcd7c61-a1be-4682-8168-f59051e470ec\" (UID: \"7bcd7c61-a1be-4682-8168-f59051e470ec\") " Oct 30 19:26:19.400625 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:26:19.400548 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bcd7c61-a1be-4682-8168-f59051e470ec-tmp\" (UniqueName: \"kubernetes.io/empty-dir/7bcd7c61-a1be-4682-8168-f59051e470ec-tmp\") pod \"7bcd7c61-a1be-4682-8168-f59051e470ec\" (UID: \"7bcd7c61-a1be-4682-8168-f59051e470ec\") " Oct 30 19:26:19.400859 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:26:19.400686 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bcd7c61-a1be-4682-8168-f59051e470ec-service-ca" pod "7bcd7c61-a1be-4682-8168-f59051e470ec" (UID: "7bcd7c61-a1be-4682-8168-f59051e470ec"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:26:19.400943 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:26:19.400915 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bcd7c61-a1be-4682-8168-f59051e470ec-tmp" pod "7bcd7c61-a1be-4682-8168-f59051e470ec" (UID: "7bcd7c61-a1be-4682-8168-f59051e470ec"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:26:19.401061 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:26:19.400962 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bcd7c61-a1be-4682-8168-f59051e470ec-console-config" pod "7bcd7c61-a1be-4682-8168-f59051e470ec" (UID: "7bcd7c61-a1be-4682-8168-f59051e470ec"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:26:19.401061 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:26:19.400988 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bcd7c61-a1be-4682-8168-f59051e470ec-trusted-ca-bundle" pod "7bcd7c61-a1be-4682-8168-f59051e470ec" (UID: "7bcd7c61-a1be-4682-8168-f59051e470ec"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:26:19.401061 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:26:19.401010 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bcd7c61-a1be-4682-8168-f59051e470ec-oauth-serving-cert" pod "7bcd7c61-a1be-4682-8168-f59051e470ec" (UID: "7bcd7c61-a1be-4682-8168-f59051e470ec"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:26:19.402547 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:26:19.402525 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bcd7c61-a1be-4682-8168-f59051e470ec-console-serving-cert" pod "7bcd7c61-a1be-4682-8168-f59051e470ec" (UID: "7bcd7c61-a1be-4682-8168-f59051e470ec"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:26:19.402699 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:26:19.402676 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bcd7c61-a1be-4682-8168-f59051e470ec-console-oauth-config" pod "7bcd7c61-a1be-4682-8168-f59051e470ec" (UID: "7bcd7c61-a1be-4682-8168-f59051e470ec"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:26:19.402760 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:26:19.402720 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bcd7c61-a1be-4682-8168-f59051e470ec-kube-api-access-krkmh" pod "7bcd7c61-a1be-4682-8168-f59051e470ec" (UID: "7bcd7c61-a1be-4682-8168-f59051e470ec"). InnerVolumeSpecName "kube-api-access-krkmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:26:19.501584 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:26:19.501554 4292 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7bcd7c61-a1be-4682-8168-f59051e470ec-trusted-ca-bundle\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:26:19.501584 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:26:19.501579 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krkmh\" (UniqueName: \"kubernetes.io/projected/7bcd7c61-a1be-4682-8168-f59051e470ec-kube-api-access-krkmh\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:26:19.501584 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:26:19.501590 4292 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7bcd7c61-a1be-4682-8168-f59051e470ec-console-config\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:26:19.501800 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:26:19.501600 4292 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7bcd7c61-a1be-4682-8168-f59051e470ec-oauth-serving-cert\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:26:19.501800 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:26:19.501610 4292 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7bcd7c61-a1be-4682-8168-f59051e470ec-tmp\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:26:19.501800 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:26:19.501619 4292 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7bcd7c61-a1be-4682-8168-f59051e470ec-console-oauth-config\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:26:19.501800 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:26:19.501627 4292 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7bcd7c61-a1be-4682-8168-f59051e470ec-service-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:26:19.501800 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:26:19.501635 4292 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7bcd7c61-a1be-4682-8168-f59051e470ec-console-serving-cert\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:26:19.944151 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:26:19.944115 4292 generic.go:356] "Generic (PLEG): container finished" podID="7bcd7c61-a1be-4682-8168-f59051e470ec" containerID="c79b6130cf60311c65eb3108b8ddccd9e3fa0117786dc2373340962a4fc5d43f" exitCode=0 Oct 30 19:26:19.944375 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:26:19.944181 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c976d4497-497d9" event={"ID":"7bcd7c61-a1be-4682-8168-f59051e470ec","Type":"ContainerDied","Data":"c79b6130cf60311c65eb3108b8ddccd9e3fa0117786dc2373340962a4fc5d43f"} Oct 30 19:26:19.944375 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:26:19.944182 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c976d4497-497d9" Oct 30 19:26:19.944375 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:26:19.944239 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c976d4497-497d9" event={"ID":"7bcd7c61-a1be-4682-8168-f59051e470ec","Type":"ContainerDied","Data":"ba084e4b99c32f681d58054896d5ba5fdec8c05fb5c81f750e04f4125e96d921"} Oct 30 19:26:19.944375 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:26:19.944258 4292 scope.go:122] "RemoveContainer" containerID="c79b6130cf60311c65eb3108b8ddccd9e3fa0117786dc2373340962a4fc5d43f" Oct 30 19:26:19.952529 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:26:19.952510 4292 scope.go:122] "RemoveContainer" containerID="c79b6130cf60311c65eb3108b8ddccd9e3fa0117786dc2373340962a4fc5d43f" Oct 30 19:26:19.952763 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:26:19.952743 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c79b6130cf60311c65eb3108b8ddccd9e3fa0117786dc2373340962a4fc5d43f\": container with ID starting with c79b6130cf60311c65eb3108b8ddccd9e3fa0117786dc2373340962a4fc5d43f not found: ID does not exist" containerID="c79b6130cf60311c65eb3108b8ddccd9e3fa0117786dc2373340962a4fc5d43f" Oct 30 19:26:19.952830 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:26:19.952773 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c79b6130cf60311c65eb3108b8ddccd9e3fa0117786dc2373340962a4fc5d43f"} err="failed to get container status \"c79b6130cf60311c65eb3108b8ddccd9e3fa0117786dc2373340962a4fc5d43f\": rpc error: code = NotFound desc = could not find container \"c79b6130cf60311c65eb3108b8ddccd9e3fa0117786dc2373340962a4fc5d43f\": container with ID starting with c79b6130cf60311c65eb3108b8ddccd9e3fa0117786dc2373340962a4fc5d43f not found: ID does not exist" Oct 30 19:26:20.099335 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:26:20.099305 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c976d4497-497d9"] Oct 30 19:26:20.146131 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:26:20.146103 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5c976d4497-497d9"] Oct 30 19:26:20.579251 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:26:20.579210 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bcd7c61-a1be-4682-8168-f59051e470ec" path="/var/lib/kubelet/pods/7bcd7c61-a1be-4682-8168-f59051e470ec/volumes" Oct 30 19:26:43.219855 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:26:43.219816 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:27:13.218791 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:27:13.218713 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:27:43.218808 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:27:43.218773 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:28:13.219305 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:28:13.219270 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:28:43.219610 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:28:43.219531 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:29:13.219242 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:29:13.219206 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:29:43.219453 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:29:43.219416 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:30:13.218864 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:30:13.218786 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:30:43.218952 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:30:43.218918 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:31:13.219577 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:31:13.219540 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:31:20.779325 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:20.779292 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c42wdb"] Oct 30 19:31:20.779831 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:20.779531 4292 cpu_manager.go:414] "containerMap: removing container" podUID="7bcd7c61-a1be-4682-8168-f59051e470ec" containerName="console" Oct 30 19:31:20.779831 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:20.779542 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bcd7c61-a1be-4682-8168-f59051e470ec" containerName="console" Oct 30 19:31:20.779831 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:20.779562 4292 cpu_manager.go:414] "containerMap: removing container" podUID="a94b1abe-f37a-439a-98e1-8e82038078ff" containerName="console" Oct 30 19:31:20.779831 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:20.779567 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a94b1abe-f37a-439a-98e1-8e82038078ff" containerName="console" Oct 30 19:31:20.779831 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:20.779609 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bcd7c61-a1be-4682-8168-f59051e470ec" containerName="console" Oct 30 19:31:20.779831 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:20.779618 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="a94b1abe-f37a-439a-98e1-8e82038078ff" containerName="console" Oct 30 19:31:20.781772 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:20.781756 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c42wdb" Oct 30 19:31:20.786265 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:20.786239 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxvlg"] Oct 30 19:31:20.788791 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:20.788723 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxvlg" Oct 30 19:31:20.832433 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:20.832404 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 19:31:20.832676 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:20.832660 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 19:31:20.836694 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:20.836675 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66fp4\" (UniqueName: \"kubernetes.io/projected/8f4e5528-6e81-4a1a-9419-d84902d6eeb8-kube-api-access-66fp4\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c42wdb\" (UID: \"8f4e5528-6e81-4a1a-9419-d84902d6eeb8\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c42wdb" Oct 30 19:31:20.836744 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:20.836723 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4dkr\" (UniqueName: \"kubernetes.io/projected/e25b3bc5-6ac1-4335-847b-8eb01dddb177-kube-api-access-x4dkr\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxvlg\" (UID: \"e25b3bc5-6ac1-4335-847b-8eb01dddb177\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxvlg" Oct 30 19:31:20.836793 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:20.836742 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e25b3bc5-6ac1-4335-847b-8eb01dddb177-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxvlg\" (UID: \"e25b3bc5-6ac1-4335-847b-8eb01dddb177\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxvlg" Oct 30 19:31:20.836862 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:20.836843 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8f4e5528-6e81-4a1a-9419-d84902d6eeb8-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c42wdb\" (UID: \"8f4e5528-6e81-4a1a-9419-d84902d6eeb8\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c42wdb" Oct 30 19:31:20.836911 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:20.836877 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f4e5528-6e81-4a1a-9419-d84902d6eeb8-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c42wdb\" (UID: \"8f4e5528-6e81-4a1a-9419-d84902d6eeb8\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c42wdb" Oct 30 19:31:20.836911 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:20.836893 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e25b3bc5-6ac1-4335-847b-8eb01dddb177-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxvlg\" (UID: \"e25b3bc5-6ac1-4335-847b-8eb01dddb177\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxvlg" Oct 30 19:31:20.892062 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:20.892025 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c42wdb"] Oct 30 19:31:20.894474 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:20.894449 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxvlg"] Oct 30 19:31:20.901764 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:20.901742 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-8jhg8\"" Oct 30 19:31:20.901764 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:20.901752 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-8j8v5\"" Oct 30 19:31:20.937266 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:20.937238 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8f4e5528-6e81-4a1a-9419-d84902d6eeb8-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c42wdb\" (UID: \"8f4e5528-6e81-4a1a-9419-d84902d6eeb8\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c42wdb" Oct 30 19:31:20.937459 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:20.937268 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f4e5528-6e81-4a1a-9419-d84902d6eeb8-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c42wdb\" (UID: \"8f4e5528-6e81-4a1a-9419-d84902d6eeb8\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c42wdb" Oct 30 19:31:20.937459 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:20.937289 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e25b3bc5-6ac1-4335-847b-8eb01dddb177-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxvlg\" (UID: \"e25b3bc5-6ac1-4335-847b-8eb01dddb177\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxvlg" Oct 30 19:31:20.937459 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:20.937322 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66fp4\" (UniqueName: \"kubernetes.io/projected/8f4e5528-6e81-4a1a-9419-d84902d6eeb8-kube-api-access-66fp4\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c42wdb\" (UID: \"8f4e5528-6e81-4a1a-9419-d84902d6eeb8\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c42wdb" Oct 30 19:31:20.937459 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:20.937378 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4dkr\" (UniqueName: \"kubernetes.io/projected/e25b3bc5-6ac1-4335-847b-8eb01dddb177-kube-api-access-x4dkr\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxvlg\" (UID: \"e25b3bc5-6ac1-4335-847b-8eb01dddb177\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxvlg" Oct 30 19:31:20.937459 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:20.937395 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e25b3bc5-6ac1-4335-847b-8eb01dddb177-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxvlg\" (UID: \"e25b3bc5-6ac1-4335-847b-8eb01dddb177\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxvlg" Oct 30 19:31:20.937827 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:20.937798 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e25b3bc5-6ac1-4335-847b-8eb01dddb177-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxvlg\" (UID: \"e25b3bc5-6ac1-4335-847b-8eb01dddb177\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxvlg" Oct 30 19:31:20.937888 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:20.937825 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8f4e5528-6e81-4a1a-9419-d84902d6eeb8-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c42wdb\" (UID: \"8f4e5528-6e81-4a1a-9419-d84902d6eeb8\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c42wdb" Oct 30 19:31:20.937888 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:20.937834 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e25b3bc5-6ac1-4335-847b-8eb01dddb177-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxvlg\" (UID: \"e25b3bc5-6ac1-4335-847b-8eb01dddb177\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxvlg" Oct 30 19:31:20.937888 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:20.937800 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f4e5528-6e81-4a1a-9419-d84902d6eeb8-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c42wdb\" (UID: \"8f4e5528-6e81-4a1a-9419-d84902d6eeb8\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c42wdb" Oct 30 19:31:20.996285 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:20.996253 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4dkr\" (UniqueName: \"kubernetes.io/projected/e25b3bc5-6ac1-4335-847b-8eb01dddb177-kube-api-access-x4dkr\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxvlg\" (UID: \"e25b3bc5-6ac1-4335-847b-8eb01dddb177\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxvlg" Oct 30 19:31:20.996434 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:20.996289 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66fp4\" (UniqueName: \"kubernetes.io/projected/8f4e5528-6e81-4a1a-9419-d84902d6eeb8-kube-api-access-66fp4\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c42wdb\" (UID: \"8f4e5528-6e81-4a1a-9419-d84902d6eeb8\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c42wdb" Oct 30 19:31:21.095494 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:21.095419 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c42wdb" Oct 30 19:31:21.100460 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:21.100434 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxvlg" Oct 30 19:31:21.147895 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:21.147174 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-qqwkz"] Oct 30 19:31:21.149748 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:21.149728 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qqwkz" Oct 30 19:31:21.202632 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:21.202600 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-qqwkz"] Oct 30 19:31:21.203614 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:21.203535 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 19:31:21.203614 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:21.203608 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 19:31:21.203795 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:21.203647 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 19:31:21.203795 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:21.203718 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 19:31:21.203795 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:21.203738 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 19:31:21.203955 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:21.203940 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-jkf5r\"" Oct 30 19:31:21.206021 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:21.205992 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 19:31:21.240046 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:21.240021 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-qqwkz\" (UID: \"ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qqwkz" Oct 30 19:31:21.240197 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:21.240058 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-qqwkz\" (UID: \"ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qqwkz" Oct 30 19:31:21.240197 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:21.240085 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-qqwkz\" (UID: \"ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qqwkz" Oct 30 19:31:21.240272 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:21.240202 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-qqwkz\" (UID: \"ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qqwkz" Oct 30 19:31:21.240272 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:21.240230 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-qqwkz\" (UID: \"ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qqwkz" Oct 30 19:31:21.240272 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:21.240256 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcrnj\" (UniqueName: \"kubernetes.io/projected/ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9-kube-api-access-pcrnj\") pod \"istiod-openshift-gateway-7876bfcb74-qqwkz\" (UID: \"ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qqwkz" Oct 30 19:31:21.240399 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:21.240275 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-qqwkz\" (UID: \"ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qqwkz" Oct 30 19:31:21.341458 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:21.341424 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-qqwkz\" (UID: \"ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qqwkz" Oct 30 19:31:21.341458 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:21.341460 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-qqwkz\" (UID: \"ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qqwkz" Oct 30 19:31:21.341706 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:21.341483 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcrnj\" (UniqueName: \"kubernetes.io/projected/ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9-kube-api-access-pcrnj\") pod \"istiod-openshift-gateway-7876bfcb74-qqwkz\" (UID: \"ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qqwkz" Oct 30 19:31:21.341706 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:21.341589 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-qqwkz\" (UID: \"ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qqwkz" Oct 30 19:31:21.341706 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:21.341654 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-qqwkz\" (UID: \"ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qqwkz" Oct 30 19:31:21.341706 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:21.341695 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-qqwkz\" (UID: \"ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qqwkz" Oct 30 19:31:21.341898 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:21.341722 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-qqwkz\" (UID: \"ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qqwkz" Oct 30 19:31:21.342246 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:21.342221 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-qqwkz\" (UID: \"ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qqwkz" Oct 30 19:31:21.343780 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:21.343756 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-qqwkz\" (UID: \"ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qqwkz" Oct 30 19:31:21.343908 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:21.343889 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-qqwkz\" (UID: \"ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qqwkz" Oct 30 19:31:21.344001 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:21.343983 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-qqwkz\" (UID: \"ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qqwkz" Oct 30 19:31:21.344056 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:21.344012 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-qqwkz\" (UID: \"ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qqwkz" Oct 30 19:31:21.399942 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:21.399870 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-qqwkz\" (UID: \"ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qqwkz" Oct 30 19:31:21.400072 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:21.400019 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcrnj\" (UniqueName: \"kubernetes.io/projected/ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9-kube-api-access-pcrnj\") pod \"istiod-openshift-gateway-7876bfcb74-qqwkz\" (UID: \"ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qqwkz" Oct 30 19:31:21.468711 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:21.468678 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qqwkz" Oct 30 19:31:21.488423 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:21.488392 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c42wdb"] Oct 30 19:31:21.494970 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:31:21.494944 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f4e5528_6e81_4a1a_9419_d84902d6eeb8.slice/crio-4c5229dd40d66e43f1df83cebf7237baf44073b3587ba382ea19a8e61d950314 WatchSource:0}: Error finding container 4c5229dd40d66e43f1df83cebf7237baf44073b3587ba382ea19a8e61d950314: Status 404 returned error can't find the container with id 4c5229dd40d66e43f1df83cebf7237baf44073b3587ba382ea19a8e61d950314 Oct 30 19:31:21.496781 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:21.496760 4292 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:31:21.520807 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:21.520779 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxvlg"] Oct 30 19:31:21.529639 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:31:21.529611 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode25b3bc5_6ac1_4335_847b_8eb01dddb177.slice/crio-0baab525db9b4768cde975df46b8979f9b57de02ec0bfebb842d0fb5f9826243 WatchSource:0}: Error finding container 0baab525db9b4768cde975df46b8979f9b57de02ec0bfebb842d0fb5f9826243: Status 404 returned error can't find the container with id 0baab525db9b4768cde975df46b8979f9b57de02ec0bfebb842d0fb5f9826243 Oct 30 19:31:21.848218 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:21.848183 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-qqwkz"] Oct 30 19:31:21.854874 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:31:21.854853 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac1a2e35_c21e_42b4_b8d2_fc0b7e7cadf9.slice/crio-e8ca3ba2e915507bf16846eb01f0c12feddbafe72e45184f649aba5436ef1245 WatchSource:0}: Error finding container e8ca3ba2e915507bf16846eb01f0c12feddbafe72e45184f649aba5436ef1245: Status 404 returned error can't find the container with id e8ca3ba2e915507bf16846eb01f0c12feddbafe72e45184f649aba5436ef1245 Oct 30 19:31:21.859795 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:21.859771 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qqwkz" event={"ID":"ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9","Type":"ContainerStarted","Data":"e8ca3ba2e915507bf16846eb01f0c12feddbafe72e45184f649aba5436ef1245"} Oct 30 19:31:21.860871 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:21.860850 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c42wdb" event={"ID":"8f4e5528-6e81-4a1a-9419-d84902d6eeb8","Type":"ContainerStarted","Data":"4c5229dd40d66e43f1df83cebf7237baf44073b3587ba382ea19a8e61d950314"} Oct 30 19:31:21.861907 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:21.861889 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxvlg" event={"ID":"e25b3bc5-6ac1-4335-847b-8eb01dddb177","Type":"ContainerStarted","Data":"0baab525db9b4768cde975df46b8979f9b57de02ec0bfebb842d0fb5f9826243"} Oct 30 19:31:26.194955 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:26.194922 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:31:26.195256 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:26.194986 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:31:26.887373 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:26.887314 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qqwkz" event={"ID":"ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9","Type":"ContainerStarted","Data":"3be37d274e06e8f57658bbfe09b63b2ca7a5dbb4b4bf1478b0db6ee3302e41aa"} Oct 30 19:31:26.887569 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:26.887462 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qqwkz" Oct 30 19:31:27.894759 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:27.894730 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qqwkz" Oct 30 19:31:28.107505 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:28.107451 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qqwkz" podStartSLOduration=2.770277454 podStartE2EDuration="7.107433518s" totalImagesPullingTime="4.337156064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:27.052112854 +0000 UTC m=+390.570428321" watchObservedRunningTime="2026-10-30 19:31:28.107433518 +0000 UTC m=+391.625748976" Oct 30 19:31:28.898294 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:28.897747 4292 generic.go:356] "Generic (PLEG): container finished" podID="8f4e5528-6e81-4a1a-9419-d84902d6eeb8" containerID="758b7eba15a87ee7e87b098bd6804531794a5d52257e8d82ddbc7c796f1cc7a7" exitCode=0 Oct 30 19:31:28.898294 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:28.897825 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c42wdb" event={"ID":"8f4e5528-6e81-4a1a-9419-d84902d6eeb8","Type":"ContainerDied","Data":"758b7eba15a87ee7e87b098bd6804531794a5d52257e8d82ddbc7c796f1cc7a7"} Oct 30 19:31:29.902784 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:29.902746 4292 generic.go:356] "Generic (PLEG): container finished" podID="e25b3bc5-6ac1-4335-847b-8eb01dddb177" containerID="1c2064150cb1dac5e952c9818818154461f1f7c7a4df96d2b116fefbee76a597" exitCode=0 Oct 30 19:31:29.903174 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:29.902797 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxvlg" event={"ID":"e25b3bc5-6ac1-4335-847b-8eb01dddb177","Type":"ContainerDied","Data":"1c2064150cb1dac5e952c9818818154461f1f7c7a4df96d2b116fefbee76a597"} Oct 30 19:31:31.912141 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:31.912109 4292 generic.go:356] "Generic (PLEG): container finished" podID="e25b3bc5-6ac1-4335-847b-8eb01dddb177" containerID="61bfcd8331e3afce302c2b15df042951168ca76d49c1c1a20821e2a09ee13cda" exitCode=0 Oct 30 19:31:31.912652 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:31.912154 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxvlg" event={"ID":"e25b3bc5-6ac1-4335-847b-8eb01dddb177","Type":"ContainerDied","Data":"61bfcd8331e3afce302c2b15df042951168ca76d49c1c1a20821e2a09ee13cda"} Oct 30 19:31:32.917877 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:32.917833 4292 generic.go:356] "Generic (PLEG): container finished" podID="8f4e5528-6e81-4a1a-9419-d84902d6eeb8" containerID="e1a600887fb5956c73647f795475be09c99e892f5d80ac84ad28c8fca1618267" exitCode=0 Oct 30 19:31:32.918370 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:32.917939 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c42wdb" event={"ID":"8f4e5528-6e81-4a1a-9419-d84902d6eeb8","Type":"ContainerDied","Data":"e1a600887fb5956c73647f795475be09c99e892f5d80ac84ad28c8fca1618267"} Oct 30 19:31:38.945208 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:38.945172 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxvlg" event={"ID":"e25b3bc5-6ac1-4335-847b-8eb01dddb177","Type":"ContainerStarted","Data":"5eb977a13027e51fe21a30c046611a7496a3ab6b0def81209a79aba2017469db"} Oct 30 19:31:39.058310 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:39.058246 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxvlg" podStartSLOduration=3.486668858 podStartE2EDuration="19.058224564s" totalImagesPullingTime="15.571555706s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:39.056932879 +0000 UTC m=+402.575248337" watchObservedRunningTime="2026-10-30 19:31:39.058224564 +0000 UTC m=+402.576540022" Oct 30 19:31:39.951240 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:39.951203 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c42wdb" event={"ID":"8f4e5528-6e81-4a1a-9419-d84902d6eeb8","Type":"ContainerStarted","Data":"a2b12021d7f410f6fd25b5f66f636f92fabf108e2415848aae6e1d70e0854967"} Oct 30 19:31:39.953263 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:39.953235 4292 generic.go:356] "Generic (PLEG): container finished" podID="e25b3bc5-6ac1-4335-847b-8eb01dddb177" containerID="5eb977a13027e51fe21a30c046611a7496a3ab6b0def81209a79aba2017469db" exitCode=0 Oct 30 19:31:39.953398 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:39.953279 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxvlg" event={"ID":"e25b3bc5-6ac1-4335-847b-8eb01dddb177","Type":"ContainerDied","Data":"5eb977a13027e51fe21a30c046611a7496a3ab6b0def81209a79aba2017469db"} Oct 30 19:31:40.093139 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:40.093042 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c42wdb" podStartSLOduration=2.9943162880000003 podStartE2EDuration="20.09302828s" totalImagesPullingTime="17.098711992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:40.092719658 +0000 UTC m=+403.611035116" watchObservedRunningTime="2026-10-30 19:31:40.09302828 +0000 UTC m=+403.611343737" Oct 30 19:31:40.958732 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:40.958702 4292 generic.go:356] "Generic (PLEG): container finished" podID="8f4e5528-6e81-4a1a-9419-d84902d6eeb8" containerID="a2b12021d7f410f6fd25b5f66f636f92fabf108e2415848aae6e1d70e0854967" exitCode=0 Oct 30 19:31:40.959136 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:40.958778 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c42wdb" event={"ID":"8f4e5528-6e81-4a1a-9419-d84902d6eeb8","Type":"ContainerDied","Data":"a2b12021d7f410f6fd25b5f66f636f92fabf108e2415848aae6e1d70e0854967"} Oct 30 19:31:41.080914 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:41.080890 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxvlg" Oct 30 19:31:41.220000 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:41.219925 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e25b3bc5-6ac1-4335-847b-8eb01dddb177-kube-api-access-x4dkr\" (UniqueName: \"kubernetes.io/projected/e25b3bc5-6ac1-4335-847b-8eb01dddb177-kube-api-access-x4dkr\") pod \"e25b3bc5-6ac1-4335-847b-8eb01dddb177\" (UID: \"e25b3bc5-6ac1-4335-847b-8eb01dddb177\") " Oct 30 19:31:41.220000 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:41.219975 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e25b3bc5-6ac1-4335-847b-8eb01dddb177-util\" (UniqueName: \"kubernetes.io/empty-dir/e25b3bc5-6ac1-4335-847b-8eb01dddb177-util\") pod \"e25b3bc5-6ac1-4335-847b-8eb01dddb177\" (UID: \"e25b3bc5-6ac1-4335-847b-8eb01dddb177\") " Oct 30 19:31:41.220175 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:41.220038 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e25b3bc5-6ac1-4335-847b-8eb01dddb177-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e25b3bc5-6ac1-4335-847b-8eb01dddb177-bundle\") pod \"e25b3bc5-6ac1-4335-847b-8eb01dddb177\" (UID: \"e25b3bc5-6ac1-4335-847b-8eb01dddb177\") " Oct 30 19:31:41.220591 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:41.220560 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e25b3bc5-6ac1-4335-847b-8eb01dddb177-bundle" pod "e25b3bc5-6ac1-4335-847b-8eb01dddb177" (UID: "e25b3bc5-6ac1-4335-847b-8eb01dddb177"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:31:41.222263 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:41.222242 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e25b3bc5-6ac1-4335-847b-8eb01dddb177-kube-api-access-x4dkr" pod "e25b3bc5-6ac1-4335-847b-8eb01dddb177" (UID: "e25b3bc5-6ac1-4335-847b-8eb01dddb177"). InnerVolumeSpecName "kube-api-access-x4dkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:31:41.224432 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:41.224409 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e25b3bc5-6ac1-4335-847b-8eb01dddb177-util" pod "e25b3bc5-6ac1-4335-847b-8eb01dddb177" (UID: "e25b3bc5-6ac1-4335-847b-8eb01dddb177"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:31:41.321384 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:41.321332 4292 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e25b3bc5-6ac1-4335-847b-8eb01dddb177-bundle\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:41.321384 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:41.321381 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4dkr\" (UniqueName: \"kubernetes.io/projected/e25b3bc5-6ac1-4335-847b-8eb01dddb177-kube-api-access-x4dkr\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:41.321384 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:41.321395 4292 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e25b3bc5-6ac1-4335-847b-8eb01dddb177-util\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:41.963696 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:41.963671 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxvlg" Oct 30 19:31:41.964204 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:41.963714 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxvlg" event={"ID":"e25b3bc5-6ac1-4335-847b-8eb01dddb177","Type":"ContainerDied","Data":"0baab525db9b4768cde975df46b8979f9b57de02ec0bfebb842d0fb5f9826243"} Oct 30 19:31:41.964204 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:41.963768 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0baab525db9b4768cde975df46b8979f9b57de02ec0bfebb842d0fb5f9826243" Oct 30 19:31:42.088791 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:42.088764 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c42wdb" Oct 30 19:31:42.229336 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:42.229256 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f4e5528-6e81-4a1a-9419-d84902d6eeb8-kube-api-access-66fp4\" (UniqueName: \"kubernetes.io/projected/8f4e5528-6e81-4a1a-9419-d84902d6eeb8-kube-api-access-66fp4\") pod \"8f4e5528-6e81-4a1a-9419-d84902d6eeb8\" (UID: \"8f4e5528-6e81-4a1a-9419-d84902d6eeb8\") " Oct 30 19:31:42.229336 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:42.229298 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f4e5528-6e81-4a1a-9419-d84902d6eeb8-util\" (UniqueName: \"kubernetes.io/empty-dir/8f4e5528-6e81-4a1a-9419-d84902d6eeb8-util\") pod \"8f4e5528-6e81-4a1a-9419-d84902d6eeb8\" (UID: \"8f4e5528-6e81-4a1a-9419-d84902d6eeb8\") " Oct 30 19:31:42.229336 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:42.229336 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f4e5528-6e81-4a1a-9419-d84902d6eeb8-bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f4e5528-6e81-4a1a-9419-d84902d6eeb8-bundle\") pod \"8f4e5528-6e81-4a1a-9419-d84902d6eeb8\" (UID: \"8f4e5528-6e81-4a1a-9419-d84902d6eeb8\") " Oct 30 19:31:42.230180 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:42.230145 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f4e5528-6e81-4a1a-9419-d84902d6eeb8-bundle" pod "8f4e5528-6e81-4a1a-9419-d84902d6eeb8" (UID: "8f4e5528-6e81-4a1a-9419-d84902d6eeb8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:31:42.231363 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:42.231317 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f4e5528-6e81-4a1a-9419-d84902d6eeb8-kube-api-access-66fp4" pod "8f4e5528-6e81-4a1a-9419-d84902d6eeb8" (UID: "8f4e5528-6e81-4a1a-9419-d84902d6eeb8"). InnerVolumeSpecName "kube-api-access-66fp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:31:42.234605 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:42.234583 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f4e5528-6e81-4a1a-9419-d84902d6eeb8-util" pod "8f4e5528-6e81-4a1a-9419-d84902d6eeb8" (UID: "8f4e5528-6e81-4a1a-9419-d84902d6eeb8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:31:42.330196 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:42.330157 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66fp4\" (UniqueName: \"kubernetes.io/projected/8f4e5528-6e81-4a1a-9419-d84902d6eeb8-kube-api-access-66fp4\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:42.330196 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:42.330187 4292 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8f4e5528-6e81-4a1a-9419-d84902d6eeb8-util\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:42.330196 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:42.330197 4292 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f4e5528-6e81-4a1a-9419-d84902d6eeb8-bundle\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:42.968580 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:42.968501 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c42wdb" event={"ID":"8f4e5528-6e81-4a1a-9419-d84902d6eeb8","Type":"ContainerDied","Data":"4c5229dd40d66e43f1df83cebf7237baf44073b3587ba382ea19a8e61d950314"} Oct 30 19:31:42.968580 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:42.968548 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c5229dd40d66e43f1df83cebf7237baf44073b3587ba382ea19a8e61d950314" Oct 30 19:31:42.968580 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:31:42.968510 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c42wdb" Oct 30 19:31:43.219841 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:31:43.219754 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:32:00.849015 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:00.848984 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-f84ddd5d6-jhwm8"] Oct 30 19:32:00.849483 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:00.849225 4292 cpu_manager.go:414] "containerMap: removing container" podUID="e25b3bc5-6ac1-4335-847b-8eb01dddb177" containerName="extract" Oct 30 19:32:00.849483 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:00.849244 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e25b3bc5-6ac1-4335-847b-8eb01dddb177" containerName="extract" Oct 30 19:32:00.849483 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:00.849265 4292 cpu_manager.go:414] "containerMap: removing container" podUID="e25b3bc5-6ac1-4335-847b-8eb01dddb177" containerName="pull" Oct 30 19:32:00.849483 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:00.849269 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e25b3bc5-6ac1-4335-847b-8eb01dddb177" containerName="pull" Oct 30 19:32:00.849483 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:00.849276 4292 cpu_manager.go:414] "containerMap: removing container" podUID="8f4e5528-6e81-4a1a-9419-d84902d6eeb8" containerName="pull" Oct 30 19:32:00.849483 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:00.849282 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f4e5528-6e81-4a1a-9419-d84902d6eeb8" containerName="pull" Oct 30 19:32:00.849483 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:00.849289 4292 cpu_manager.go:414] "containerMap: removing container" podUID="8f4e5528-6e81-4a1a-9419-d84902d6eeb8" containerName="util" Oct 30 19:32:00.849483 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:00.849293 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f4e5528-6e81-4a1a-9419-d84902d6eeb8" containerName="util" Oct 30 19:32:00.849483 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:00.849300 4292 cpu_manager.go:414] "containerMap: removing container" podUID="e25b3bc5-6ac1-4335-847b-8eb01dddb177" containerName="util" Oct 30 19:32:00.849483 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:00.849305 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e25b3bc5-6ac1-4335-847b-8eb01dddb177" containerName="util" Oct 30 19:32:00.849483 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:00.849310 4292 cpu_manager.go:414] "containerMap: removing container" podUID="8f4e5528-6e81-4a1a-9419-d84902d6eeb8" containerName="extract" Oct 30 19:32:00.849483 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:00.849315 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f4e5528-6e81-4a1a-9419-d84902d6eeb8" containerName="extract" Oct 30 19:32:00.849483 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:00.849368 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f4e5528-6e81-4a1a-9419-d84902d6eeb8" containerName="extract" Oct 30 19:32:00.849483 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:00.849379 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="e25b3bc5-6ac1-4335-847b-8eb01dddb177" containerName="extract" Oct 30 19:32:00.853897 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:00.853879 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f84ddd5d6-jhwm8" Oct 30 19:32:00.960492 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:00.960457 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 19:32:00.960492 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:00.960469 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 19:32:00.960729 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:00.960496 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 19:32:00.960729 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:00.960490 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 19:32:00.960729 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:00.960503 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-dvbx8\"" Oct 30 19:32:00.960729 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:00.960575 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 19:32:00.961802 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:00.961782 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-f84ddd5d6-jhwm8"] Oct 30 19:32:00.968104 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:00.968084 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hxjw\" (UniqueName: \"kubernetes.io/projected/18f533d2-b352-408a-b443-3d76c256e9e1-kube-api-access-9hxjw\") pod \"opendatahub-operator-controller-manager-f84ddd5d6-jhwm8\" (UID: \"18f533d2-b352-408a-b443-3d76c256e9e1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f84ddd5d6-jhwm8" Oct 30 19:32:00.968201 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:00.968155 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/18f533d2-b352-408a-b443-3d76c256e9e1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f84ddd5d6-jhwm8\" (UID: \"18f533d2-b352-408a-b443-3d76c256e9e1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f84ddd5d6-jhwm8" Oct 30 19:32:00.968262 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:00.968240 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/18f533d2-b352-408a-b443-3d76c256e9e1-manifests\") pod \"opendatahub-operator-controller-manager-f84ddd5d6-jhwm8\" (UID: \"18f533d2-b352-408a-b443-3d76c256e9e1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f84ddd5d6-jhwm8" Oct 30 19:32:00.968319 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:00.968272 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/18f533d2-b352-408a-b443-3d76c256e9e1-tmp\") pod \"opendatahub-operator-controller-manager-f84ddd5d6-jhwm8\" (UID: \"18f533d2-b352-408a-b443-3d76c256e9e1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f84ddd5d6-jhwm8" Oct 30 19:32:00.968319 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:00.968302 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/18f533d2-b352-408a-b443-3d76c256e9e1-metrics-cert\") pod \"opendatahub-operator-controller-manager-f84ddd5d6-jhwm8\" (UID: \"18f533d2-b352-408a-b443-3d76c256e9e1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f84ddd5d6-jhwm8" Oct 30 19:32:00.968409 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:00.968342 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/18f533d2-b352-408a-b443-3d76c256e9e1-webhook-cert\") pod \"opendatahub-operator-controller-manager-f84ddd5d6-jhwm8\" (UID: \"18f533d2-b352-408a-b443-3d76c256e9e1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f84ddd5d6-jhwm8" Oct 30 19:32:01.069173 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:01.069139 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/18f533d2-b352-408a-b443-3d76c256e9e1-tmp\") pod \"opendatahub-operator-controller-manager-f84ddd5d6-jhwm8\" (UID: \"18f533d2-b352-408a-b443-3d76c256e9e1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f84ddd5d6-jhwm8" Oct 30 19:32:01.069173 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:01.069176 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/18f533d2-b352-408a-b443-3d76c256e9e1-metrics-cert\") pod \"opendatahub-operator-controller-manager-f84ddd5d6-jhwm8\" (UID: \"18f533d2-b352-408a-b443-3d76c256e9e1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f84ddd5d6-jhwm8" Oct 30 19:32:01.069430 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:01.069193 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/18f533d2-b352-408a-b443-3d76c256e9e1-webhook-cert\") pod \"opendatahub-operator-controller-manager-f84ddd5d6-jhwm8\" (UID: \"18f533d2-b352-408a-b443-3d76c256e9e1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f84ddd5d6-jhwm8" Oct 30 19:32:01.069430 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:01.069217 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hxjw\" (UniqueName: \"kubernetes.io/projected/18f533d2-b352-408a-b443-3d76c256e9e1-kube-api-access-9hxjw\") pod \"opendatahub-operator-controller-manager-f84ddd5d6-jhwm8\" (UID: \"18f533d2-b352-408a-b443-3d76c256e9e1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f84ddd5d6-jhwm8" Oct 30 19:32:01.069430 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:01.069270 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/18f533d2-b352-408a-b443-3d76c256e9e1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f84ddd5d6-jhwm8\" (UID: \"18f533d2-b352-408a-b443-3d76c256e9e1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f84ddd5d6-jhwm8" Oct 30 19:32:01.069430 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:01.069302 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/18f533d2-b352-408a-b443-3d76c256e9e1-manifests\") pod \"opendatahub-operator-controller-manager-f84ddd5d6-jhwm8\" (UID: \"18f533d2-b352-408a-b443-3d76c256e9e1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f84ddd5d6-jhwm8" Oct 30 19:32:01.069680 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:01.069654 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/18f533d2-b352-408a-b443-3d76c256e9e1-manifests\") pod \"opendatahub-operator-controller-manager-f84ddd5d6-jhwm8\" (UID: \"18f533d2-b352-408a-b443-3d76c256e9e1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f84ddd5d6-jhwm8" Oct 30 19:32:01.071495 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:01.071469 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/18f533d2-b352-408a-b443-3d76c256e9e1-tmp\") pod \"opendatahub-operator-controller-manager-f84ddd5d6-jhwm8\" (UID: \"18f533d2-b352-408a-b443-3d76c256e9e1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f84ddd5d6-jhwm8" Oct 30 19:32:01.071777 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:01.071757 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/18f533d2-b352-408a-b443-3d76c256e9e1-webhook-cert\") pod \"opendatahub-operator-controller-manager-f84ddd5d6-jhwm8\" (UID: \"18f533d2-b352-408a-b443-3d76c256e9e1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f84ddd5d6-jhwm8" Oct 30 19:32:01.071841 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:01.071818 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/18f533d2-b352-408a-b443-3d76c256e9e1-metrics-cert\") pod \"opendatahub-operator-controller-manager-f84ddd5d6-jhwm8\" (UID: \"18f533d2-b352-408a-b443-3d76c256e9e1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f84ddd5d6-jhwm8" Oct 30 19:32:01.071897 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:01.071853 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/18f533d2-b352-408a-b443-3d76c256e9e1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f84ddd5d6-jhwm8\" (UID: \"18f533d2-b352-408a-b443-3d76c256e9e1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f84ddd5d6-jhwm8" Oct 30 19:32:01.133062 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:01.133004 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hxjw\" (UniqueName: \"kubernetes.io/projected/18f533d2-b352-408a-b443-3d76c256e9e1-kube-api-access-9hxjw\") pod \"opendatahub-operator-controller-manager-f84ddd5d6-jhwm8\" (UID: \"18f533d2-b352-408a-b443-3d76c256e9e1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f84ddd5d6-jhwm8" Oct 30 19:32:01.169930 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:01.169903 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f84ddd5d6-jhwm8" Oct 30 19:32:01.681170 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:01.681136 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-f84ddd5d6-jhwm8"] Oct 30 19:32:01.686205 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:32:01.686168 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18f533d2_b352_408a_b443_3d76c256e9e1.slice/crio-50e9185ab0bfa13160f8590c74c27160d0f7fc77a32d8a63bd18c4fbfa957c1d WatchSource:0}: Error finding container 50e9185ab0bfa13160f8590c74c27160d0f7fc77a32d8a63bd18c4fbfa957c1d: Status 404 returned error can't find the container with id 50e9185ab0bfa13160f8590c74c27160d0f7fc77a32d8a63bd18c4fbfa957c1d Oct 30 19:32:02.037469 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:02.037436 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f84ddd5d6-jhwm8" event={"ID":"18f533d2-b352-408a-b443-3d76c256e9e1","Type":"ContainerStarted","Data":"50e9185ab0bfa13160f8590c74c27160d0f7fc77a32d8a63bd18c4fbfa957c1d"} Oct 30 19:32:06.054091 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:06.054057 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f84ddd5d6-jhwm8" event={"ID":"18f533d2-b352-408a-b443-3d76c256e9e1","Type":"ContainerStarted","Data":"b3ab2413c079f4f21ba5be237117501b8700ada2f3a6aa66f1372175144685a4"} Oct 30 19:32:07.058856 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:07.058821 4292 generic.go:356] "Generic (PLEG): container finished" podID="18f533d2-b352-408a-b443-3d76c256e9e1" containerID="b3ab2413c079f4f21ba5be237117501b8700ada2f3a6aa66f1372175144685a4" exitCode=0 Oct 30 19:32:07.059228 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:07.058902 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f84ddd5d6-jhwm8" event={"ID":"18f533d2-b352-408a-b443-3d76c256e9e1","Type":"ContainerDied","Data":"b3ab2413c079f4f21ba5be237117501b8700ada2f3a6aa66f1372175144685a4"} Oct 30 19:32:08.063864 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:08.063827 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f84ddd5d6-jhwm8" event={"ID":"18f533d2-b352-408a-b443-3d76c256e9e1","Type":"ContainerStarted","Data":"d8764dcc322efb24bd485f12d6d5d43e29e4a2cdc1cfd40e9d5ff7f167eac52d"} Oct 30 19:32:08.064233 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:08.063946 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f84ddd5d6-jhwm8" Oct 30 19:32:08.300830 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:08.300780 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f84ddd5d6-jhwm8" podStartSLOduration=2.702675045 podStartE2EDuration="8.30076476s" totalImagesPullingTime="5.175895839s" totalInitContainerRuntime="422.193876ms" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:08.253665765 +0000 UTC m=+431.771981222" watchObservedRunningTime="2026-10-30 19:32:08.30076476 +0000 UTC m=+431.819080217" Oct 30 19:32:13.218918 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:32:13.218875 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:32:19.069534 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:19.069502 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f84ddd5d6-jhwm8" Oct 30 19:32:41.085840 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:41.085799 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5fc9b9cc5d-nft4n"] Oct 30 19:32:41.092801 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:41.092779 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5fc9b9cc5d-nft4n" Oct 30 19:32:41.144744 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:41.144709 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5fc9b9cc5d-nft4n"] Oct 30 19:32:41.148223 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:41.148195 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e8dccba-2e77-4679-bed8-4ee069155583-tmp\") pod \"kube-auth-proxy-5fc9b9cc5d-nft4n\" (UID: \"9e8dccba-2e77-4679-bed8-4ee069155583\") " pod="openshift-ingress/kube-auth-proxy-5fc9b9cc5d-nft4n" Oct 30 19:32:41.148390 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:41.148252 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g44xt\" (UniqueName: \"kubernetes.io/projected/9e8dccba-2e77-4679-bed8-4ee069155583-kube-api-access-g44xt\") pod \"kube-auth-proxy-5fc9b9cc5d-nft4n\" (UID: \"9e8dccba-2e77-4679-bed8-4ee069155583\") " pod="openshift-ingress/kube-auth-proxy-5fc9b9cc5d-nft4n" Oct 30 19:32:41.148390 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:41.148317 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e8dccba-2e77-4679-bed8-4ee069155583-tls-certs\") pod \"kube-auth-proxy-5fc9b9cc5d-nft4n\" (UID: \"9e8dccba-2e77-4679-bed8-4ee069155583\") " pod="openshift-ingress/kube-auth-proxy-5fc9b9cc5d-nft4n" Oct 30 19:32:41.189832 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:41.189800 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 19:32:41.189832 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:41.189826 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 19:32:41.190043 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:41.189915 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-5mtsh\"" Oct 30 19:32:41.248840 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:41.248800 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g44xt\" (UniqueName: \"kubernetes.io/projected/9e8dccba-2e77-4679-bed8-4ee069155583-kube-api-access-g44xt\") pod \"kube-auth-proxy-5fc9b9cc5d-nft4n\" (UID: \"9e8dccba-2e77-4679-bed8-4ee069155583\") " pod="openshift-ingress/kube-auth-proxy-5fc9b9cc5d-nft4n" Oct 30 19:32:41.248840 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:41.248843 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e8dccba-2e77-4679-bed8-4ee069155583-tls-certs\") pod \"kube-auth-proxy-5fc9b9cc5d-nft4n\" (UID: \"9e8dccba-2e77-4679-bed8-4ee069155583\") " pod="openshift-ingress/kube-auth-proxy-5fc9b9cc5d-nft4n" Oct 30 19:32:41.249052 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:41.248876 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e8dccba-2e77-4679-bed8-4ee069155583-tmp\") pod \"kube-auth-proxy-5fc9b9cc5d-nft4n\" (UID: \"9e8dccba-2e77-4679-bed8-4ee069155583\") " pod="openshift-ingress/kube-auth-proxy-5fc9b9cc5d-nft4n" Oct 30 19:32:41.251135 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:41.251103 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e8dccba-2e77-4679-bed8-4ee069155583-tmp\") pod \"kube-auth-proxy-5fc9b9cc5d-nft4n\" (UID: \"9e8dccba-2e77-4679-bed8-4ee069155583\") " pod="openshift-ingress/kube-auth-proxy-5fc9b9cc5d-nft4n" Oct 30 19:32:41.251326 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:41.251310 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e8dccba-2e77-4679-bed8-4ee069155583-tls-certs\") pod \"kube-auth-proxy-5fc9b9cc5d-nft4n\" (UID: \"9e8dccba-2e77-4679-bed8-4ee069155583\") " pod="openshift-ingress/kube-auth-proxy-5fc9b9cc5d-nft4n" Oct 30 19:32:41.307135 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:41.307103 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g44xt\" (UniqueName: \"kubernetes.io/projected/9e8dccba-2e77-4679-bed8-4ee069155583-kube-api-access-g44xt\") pod \"kube-auth-proxy-5fc9b9cc5d-nft4n\" (UID: \"9e8dccba-2e77-4679-bed8-4ee069155583\") " pod="openshift-ingress/kube-auth-proxy-5fc9b9cc5d-nft4n" Oct 30 19:32:41.403263 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:41.403186 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5fc9b9cc5d-nft4n" Oct 30 19:32:41.775789 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:41.775760 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5fc9b9cc5d-nft4n"] Oct 30 19:32:41.783444 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:32:41.783417 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e8dccba_2e77_4679_bed8_4ee069155583.slice/crio-b17f7b978f68cae6e41de356cccb9f705db49b7aba9aec10eaab8609c19c6e30 WatchSource:0}: Error finding container b17f7b978f68cae6e41de356cccb9f705db49b7aba9aec10eaab8609c19c6e30: Status 404 returned error can't find the container with id b17f7b978f68cae6e41de356cccb9f705db49b7aba9aec10eaab8609c19c6e30 Oct 30 19:32:42.174669 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:42.174590 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5fc9b9cc5d-nft4n" event={"ID":"9e8dccba-2e77-4679-bed8-4ee069155583","Type":"ContainerStarted","Data":"b17f7b978f68cae6e41de356cccb9f705db49b7aba9aec10eaab8609c19c6e30"} Oct 30 19:32:43.229224 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:32:43.229182 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:32:45.188259 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:45.188216 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5fc9b9cc5d-nft4n" event={"ID":"9e8dccba-2e77-4679-bed8-4ee069155583","Type":"ContainerStarted","Data":"a72efaa5eab92206bfa3289f4d5afec4cb69fb6096f7c24b4972b5947ab8105d"} Oct 30 19:32:45.300538 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:45.300484 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5fc9b9cc5d-nft4n" podStartSLOduration=1.496026621 podStartE2EDuration="4.300470703s" totalImagesPullingTime="2.804444082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:45.300113506 +0000 UTC m=+468.818428967" watchObservedRunningTime="2026-10-30 19:32:45.300470703 +0000 UTC m=+468.818786160" Oct 30 19:32:45.580328 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:45.580296 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-6vdd2"] Oct 30 19:32:45.583532 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:45.583506 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-6vdd2" Oct 30 19:32:45.633433 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:45.633402 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 19:32:45.633620 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:45.633564 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 19:32:45.633710 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:45.633691 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 19:32:45.633710 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:45.633701 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-rq9w4\"" Oct 30 19:32:45.680014 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:45.679985 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j9ht\" (UniqueName: \"kubernetes.io/projected/c0d5ddf7-16c6-4ec0-933a-e95a8e5b7065-kube-api-access-8j9ht\") pod \"limitador-limitador-74dfcfb894-6vdd2\" (UID: \"c0d5ddf7-16c6-4ec0-933a-e95a8e5b7065\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-6vdd2" Oct 30 19:32:45.680168 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:45.680035 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c0d5ddf7-16c6-4ec0-933a-e95a8e5b7065-config-file\") pod \"limitador-limitador-74dfcfb894-6vdd2\" (UID: \"c0d5ddf7-16c6-4ec0-933a-e95a8e5b7065\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-6vdd2" Oct 30 19:32:45.681048 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:45.681028 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-6vdd2"] Oct 30 19:32:45.780566 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:45.780533 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j9ht\" (UniqueName: \"kubernetes.io/projected/c0d5ddf7-16c6-4ec0-933a-e95a8e5b7065-kube-api-access-8j9ht\") pod \"limitador-limitador-74dfcfb894-6vdd2\" (UID: \"c0d5ddf7-16c6-4ec0-933a-e95a8e5b7065\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-6vdd2" Oct 30 19:32:45.780728 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:45.780610 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c0d5ddf7-16c6-4ec0-933a-e95a8e5b7065-config-file\") pod \"limitador-limitador-74dfcfb894-6vdd2\" (UID: \"c0d5ddf7-16c6-4ec0-933a-e95a8e5b7065\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-6vdd2" Oct 30 19:32:45.781311 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:45.781293 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c0d5ddf7-16c6-4ec0-933a-e95a8e5b7065-config-file\") pod \"limitador-limitador-74dfcfb894-6vdd2\" (UID: \"c0d5ddf7-16c6-4ec0-933a-e95a8e5b7065\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-6vdd2" Oct 30 19:32:45.809114 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:45.809080 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-6vdd2"] Oct 30 19:32:45.839262 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:45.839192 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j9ht\" (UniqueName: \"kubernetes.io/projected/c0d5ddf7-16c6-4ec0-933a-e95a8e5b7065-kube-api-access-8j9ht\") pod \"limitador-limitador-74dfcfb894-6vdd2\" (UID: \"c0d5ddf7-16c6-4ec0-933a-e95a8e5b7065\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-6vdd2" Oct 30 19:32:45.895482 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:45.895450 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-6vdd2" Oct 30 19:32:46.263850 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:46.263772 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-6vdd2"] Oct 30 19:32:46.270859 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:32:46.270834 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0d5ddf7_16c6_4ec0_933a_e95a8e5b7065.slice/crio-e5f3a627b5483b339d37e3dc5894bc25cc4ffa4a37d2deb56f32893f2811b4e1 WatchSource:0}: Error finding container e5f3a627b5483b339d37e3dc5894bc25cc4ffa4a37d2deb56f32893f2811b4e1: Status 404 returned error can't find the container with id e5f3a627b5483b339d37e3dc5894bc25cc4ffa4a37d2deb56f32893f2811b4e1 Oct 30 19:32:47.196526 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:47.196492 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-6vdd2" event={"ID":"c0d5ddf7-16c6-4ec0-933a-e95a8e5b7065","Type":"ContainerStarted","Data":"e5f3a627b5483b339d37e3dc5894bc25cc4ffa4a37d2deb56f32893f2811b4e1"} Oct 30 19:32:50.306072 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:50.306041 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-q9kwv"] Oct 30 19:32:50.309501 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:50.309477 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-q9kwv" Oct 30 19:32:50.368075 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:50.368043 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Oct 30 19:32:50.369823 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:50.369743 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-q9kwv"] Oct 30 19:32:50.405847 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:50.405813 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Oct 30 19:32:50.414212 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:50.414178 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/90740ed8-ad1f-42ac-8e4c-4db19bde978a-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-q9kwv\" (UID: \"90740ed8-ad1f-42ac-8e4c-4db19bde978a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-q9kwv" Oct 30 19:32:50.414212 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:50.414213 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvx7v\" (UniqueName: \"kubernetes.io/projected/90740ed8-ad1f-42ac-8e4c-4db19bde978a-kube-api-access-bvx7v\") pod \"kuadrant-console-plugin-66c4fb575d-q9kwv\" (UID: \"90740ed8-ad1f-42ac-8e4c-4db19bde978a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-q9kwv" Oct 30 19:32:50.414395 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:50.414255 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/90740ed8-ad1f-42ac-8e4c-4db19bde978a-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-q9kwv\" (UID: \"90740ed8-ad1f-42ac-8e4c-4db19bde978a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-q9kwv" Oct 30 19:32:50.515221 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:50.515197 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/90740ed8-ad1f-42ac-8e4c-4db19bde978a-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-q9kwv\" (UID: \"90740ed8-ad1f-42ac-8e4c-4db19bde978a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-q9kwv" Oct 30 19:32:50.515330 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:50.515272 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/90740ed8-ad1f-42ac-8e4c-4db19bde978a-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-q9kwv\" (UID: \"90740ed8-ad1f-42ac-8e4c-4db19bde978a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-q9kwv" Oct 30 19:32:50.515330 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:50.515301 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvx7v\" (UniqueName: \"kubernetes.io/projected/90740ed8-ad1f-42ac-8e4c-4db19bde978a-kube-api-access-bvx7v\") pod \"kuadrant-console-plugin-66c4fb575d-q9kwv\" (UID: \"90740ed8-ad1f-42ac-8e4c-4db19bde978a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-q9kwv" Oct 30 19:32:50.515938 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:50.515912 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/90740ed8-ad1f-42ac-8e4c-4db19bde978a-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-q9kwv\" (UID: \"90740ed8-ad1f-42ac-8e4c-4db19bde978a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-q9kwv" Oct 30 19:32:50.517804 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:50.517776 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/90740ed8-ad1f-42ac-8e4c-4db19bde978a-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-q9kwv\" (UID: \"90740ed8-ad1f-42ac-8e4c-4db19bde978a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-q9kwv" Oct 30 19:32:50.573214 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:50.573142 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvx7v\" (UniqueName: \"kubernetes.io/projected/90740ed8-ad1f-42ac-8e4c-4db19bde978a-kube-api-access-bvx7v\") pod \"kuadrant-console-plugin-66c4fb575d-q9kwv\" (UID: \"90740ed8-ad1f-42ac-8e4c-4db19bde978a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-q9kwv" Oct 30 19:32:50.634610 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:50.634576 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-q9kwv" Oct 30 19:32:51.005669 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:51.005590 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-q9kwv"] Oct 30 19:32:51.012714 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:32:51.012689 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod90740ed8_ad1f_42ac_8e4c_4db19bde978a.slice/crio-b419fd638664abb3377e366384373df0df108683a763ed90edb18ea0f1acc4da WatchSource:0}: Error finding container b419fd638664abb3377e366384373df0df108683a763ed90edb18ea0f1acc4da: Status 404 returned error can't find the container with id b419fd638664abb3377e366384373df0df108683a763ed90edb18ea0f1acc4da Oct 30 19:32:51.212501 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:51.212469 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-q9kwv" event={"ID":"90740ed8-ad1f-42ac-8e4c-4db19bde978a","Type":"ContainerStarted","Data":"b419fd638664abb3377e366384373df0df108683a763ed90edb18ea0f1acc4da"} Oct 30 19:32:51.213737 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:51.213715 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-6vdd2" event={"ID":"c0d5ddf7-16c6-4ec0-933a-e95a8e5b7065","Type":"ContainerStarted","Data":"3e0055bc1359fbc86f49a9c8d38d174f517fbe265b8fe72e24406f95ae811242"} Oct 30 19:32:51.213875 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:51.213855 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-6vdd2" Oct 30 19:32:51.326088 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:51.326035 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-6vdd2" podStartSLOduration=2.133601642 podStartE2EDuration="6.326020816s" totalImagesPullingTime="4.192419174s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:51.324940429 +0000 UTC m=+474.843255887" watchObservedRunningTime="2026-10-30 19:32:51.326020816 +0000 UTC m=+474.844336267" Oct 30 19:32:53.014210 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:53.014179 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-dshrx"] Oct 30 19:32:53.018472 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:53.018450 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dshrx" Oct 30 19:32:53.033829 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:53.033805 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/896347de-fc5a-480a-a436-0d2f26b0b76c-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-dshrx\" (UID: \"896347de-fc5a-480a-a436-0d2f26b0b76c\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dshrx" Oct 30 19:32:53.033938 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:53.033835 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7fbx\" (UniqueName: \"kubernetes.io/projected/896347de-fc5a-480a-a436-0d2f26b0b76c-kube-api-access-m7fbx\") pod \"kserve-module-controller-manager-7566cf5fc8-dshrx\" (UID: \"896347de-fc5a-480a-a436-0d2f26b0b76c\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dshrx" Oct 30 19:32:53.033938 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:53.033856 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/896347de-fc5a-480a-a436-0d2f26b0b76c-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-dshrx\" (UID: \"896347de-fc5a-480a-a436-0d2f26b0b76c\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dshrx" Oct 30 19:32:53.033938 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:53.033914 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/896347de-fc5a-480a-a436-0d2f26b0b76c-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-dshrx\" (UID: \"896347de-fc5a-480a-a436-0d2f26b0b76c\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dshrx" Oct 30 19:32:53.070146 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:53.070115 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-pv5l5\"" Oct 30 19:32:53.116925 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:53.116892 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-dshrx"] Oct 30 19:32:53.133762 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:53.133731 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 19:32:53.133889 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:53.133811 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 19:32:53.135094 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:53.135076 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/896347de-fc5a-480a-a436-0d2f26b0b76c-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-dshrx\" (UID: \"896347de-fc5a-480a-a436-0d2f26b0b76c\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dshrx" Oct 30 19:32:53.135138 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:53.135102 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7fbx\" (UniqueName: \"kubernetes.io/projected/896347de-fc5a-480a-a436-0d2f26b0b76c-kube-api-access-m7fbx\") pod \"kserve-module-controller-manager-7566cf5fc8-dshrx\" (UID: \"896347de-fc5a-480a-a436-0d2f26b0b76c\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dshrx" Oct 30 19:32:53.135138 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:53.135120 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/896347de-fc5a-480a-a436-0d2f26b0b76c-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-dshrx\" (UID: \"896347de-fc5a-480a-a436-0d2f26b0b76c\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dshrx" Oct 30 19:32:53.135233 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:53.135139 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/896347de-fc5a-480a-a436-0d2f26b0b76c-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-dshrx\" (UID: \"896347de-fc5a-480a-a436-0d2f26b0b76c\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dshrx" Oct 30 19:32:53.135479 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:53.135459 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/896347de-fc5a-480a-a436-0d2f26b0b76c-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-dshrx\" (UID: \"896347de-fc5a-480a-a436-0d2f26b0b76c\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dshrx" Oct 30 19:32:53.135548 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:53.135503 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/896347de-fc5a-480a-a436-0d2f26b0b76c-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-dshrx\" (UID: \"896347de-fc5a-480a-a436-0d2f26b0b76c\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dshrx" Oct 30 19:32:53.226943 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:53.226918 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Oct 30 19:32:53.229190 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:53.229161 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7fbx\" (UniqueName: \"kubernetes.io/projected/896347de-fc5a-480a-a436-0d2f26b0b76c-kube-api-access-m7fbx\") pod \"kserve-module-controller-manager-7566cf5fc8-dshrx\" (UID: \"896347de-fc5a-480a-a436-0d2f26b0b76c\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dshrx" Oct 30 19:32:53.236718 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:53.236687 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/896347de-fc5a-480a-a436-0d2f26b0b76c-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-dshrx\" (UID: \"896347de-fc5a-480a-a436-0d2f26b0b76c\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dshrx" Oct 30 19:32:53.331092 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:53.331056 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dshrx" Oct 30 19:32:53.770569 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:53.770531 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-dshrx"] Oct 30 19:32:53.778389 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:32:53.777912 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod896347de_fc5a_480a_a436_0d2f26b0b76c.slice/crio-fe414388592d53a51e9505c2e904755c8a781c2dbad4719a2292b7d7ee5eb01e WatchSource:0}: Error finding container fe414388592d53a51e9505c2e904755c8a781c2dbad4719a2292b7d7ee5eb01e: Status 404 returned error can't find the container with id fe414388592d53a51e9505c2e904755c8a781c2dbad4719a2292b7d7ee5eb01e Oct 30 19:32:54.232620 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:54.232521 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dshrx" event={"ID":"896347de-fc5a-480a-a436-0d2f26b0b76c","Type":"ContainerStarted","Data":"fe414388592d53a51e9505c2e904755c8a781c2dbad4719a2292b7d7ee5eb01e"} Oct 30 19:32:56.243343 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:56.243308 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dshrx" event={"ID":"896347de-fc5a-480a-a436-0d2f26b0b76c","Type":"ContainerStarted","Data":"c6e1b0f162f8ca84f324c7b0c58d61955fd8e221eae2ecb3a4f2ed6c7898c79e"} Oct 30 19:32:56.243786 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:56.243442 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dshrx" Oct 30 19:32:56.404658 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:56.404565 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dshrx" podStartSLOduration=2.128249768 podStartE2EDuration="4.404550554s" totalImagesPullingTime="2.276300786s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:56.403138857 +0000 UTC m=+479.921454334" watchObservedRunningTime="2026-10-30 19:32:56.404550554 +0000 UTC m=+479.922866010" Oct 30 19:32:58.253087 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:58.253043 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-q9kwv" event={"ID":"90740ed8-ad1f-42ac-8e4c-4db19bde978a","Type":"ContainerStarted","Data":"f2e2e0ed99c310e4fb79293b7577f1517497650dc76d022f82fc978866a23a37"} Oct 30 19:32:58.363698 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:32:58.363647 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-q9kwv" podStartSLOduration=1.9419353639999999 podStartE2EDuration="8.36363353s" totalImagesPullingTime="6.421698166s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:58.362842811 +0000 UTC m=+481.881158269" watchObservedRunningTime="2026-10-30 19:32:58.36363353 +0000 UTC m=+481.881948987" Oct 30 19:33:02.218790 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:02.218759 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-6vdd2" Oct 30 19:33:05.073216 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:05.073178 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-6vdd2"] Oct 30 19:33:05.073678 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:05.073448 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-6vdd2" podUID="c0d5ddf7-16c6-4ec0-933a-e95a8e5b7065" containerName="limitador" containerID="cri-o://3e0055bc1359fbc86f49a9c8d38d174f517fbe265b8fe72e24406f95ae811242" gracePeriod=30 Oct 30 19:33:05.610806 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:05.610783 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-6vdd2" Oct 30 19:33:05.736679 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:05.736603 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0d5ddf7-16c6-4ec0-933a-e95a8e5b7065-config-file\" (UniqueName: \"kubernetes.io/configmap/c0d5ddf7-16c6-4ec0-933a-e95a8e5b7065-config-file\") pod \"c0d5ddf7-16c6-4ec0-933a-e95a8e5b7065\" (UID: \"c0d5ddf7-16c6-4ec0-933a-e95a8e5b7065\") " Oct 30 19:33:05.736822 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:05.736705 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0d5ddf7-16c6-4ec0-933a-e95a8e5b7065-kube-api-access-8j9ht\" (UniqueName: \"kubernetes.io/projected/c0d5ddf7-16c6-4ec0-933a-e95a8e5b7065-kube-api-access-8j9ht\") pod \"c0d5ddf7-16c6-4ec0-933a-e95a8e5b7065\" (UID: \"c0d5ddf7-16c6-4ec0-933a-e95a8e5b7065\") " Oct 30 19:33:05.736955 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:05.736931 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0d5ddf7-16c6-4ec0-933a-e95a8e5b7065-config-file" pod "c0d5ddf7-16c6-4ec0-933a-e95a8e5b7065" (UID: "c0d5ddf7-16c6-4ec0-933a-e95a8e5b7065"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:33:05.738963 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:05.738938 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0d5ddf7-16c6-4ec0-933a-e95a8e5b7065-kube-api-access-8j9ht" pod "c0d5ddf7-16c6-4ec0-933a-e95a8e5b7065" (UID: "c0d5ddf7-16c6-4ec0-933a-e95a8e5b7065"). InnerVolumeSpecName "kube-api-access-8j9ht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:33:05.838141 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:05.838109 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8j9ht\" (UniqueName: \"kubernetes.io/projected/c0d5ddf7-16c6-4ec0-933a-e95a8e5b7065-kube-api-access-8j9ht\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:05.838141 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:05.838136 4292 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c0d5ddf7-16c6-4ec0-933a-e95a8e5b7065-config-file\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:06.286986 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:06.286950 4292 generic.go:356] "Generic (PLEG): container finished" podID="c0d5ddf7-16c6-4ec0-933a-e95a8e5b7065" containerID="3e0055bc1359fbc86f49a9c8d38d174f517fbe265b8fe72e24406f95ae811242" exitCode=0 Oct 30 19:33:06.287503 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:06.287007 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-6vdd2" Oct 30 19:33:06.287503 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:06.287035 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-6vdd2" event={"ID":"c0d5ddf7-16c6-4ec0-933a-e95a8e5b7065","Type":"ContainerDied","Data":"3e0055bc1359fbc86f49a9c8d38d174f517fbe265b8fe72e24406f95ae811242"} Oct 30 19:33:06.287503 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:06.287092 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-6vdd2" event={"ID":"c0d5ddf7-16c6-4ec0-933a-e95a8e5b7065","Type":"ContainerDied","Data":"e5f3a627b5483b339d37e3dc5894bc25cc4ffa4a37d2deb56f32893f2811b4e1"} Oct 30 19:33:06.287503 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:06.287116 4292 scope.go:122] "RemoveContainer" containerID="3e0055bc1359fbc86f49a9c8d38d174f517fbe265b8fe72e24406f95ae811242" Oct 30 19:33:06.295006 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:06.294928 4292 scope.go:122] "RemoveContainer" containerID="3e0055bc1359fbc86f49a9c8d38d174f517fbe265b8fe72e24406f95ae811242" Oct 30 19:33:06.295218 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:33:06.295197 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e0055bc1359fbc86f49a9c8d38d174f517fbe265b8fe72e24406f95ae811242\": container with ID starting with 3e0055bc1359fbc86f49a9c8d38d174f517fbe265b8fe72e24406f95ae811242 not found: ID does not exist" containerID="3e0055bc1359fbc86f49a9c8d38d174f517fbe265b8fe72e24406f95ae811242" Oct 30 19:33:06.295292 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:06.295225 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e0055bc1359fbc86f49a9c8d38d174f517fbe265b8fe72e24406f95ae811242"} err="failed to get container status \"3e0055bc1359fbc86f49a9c8d38d174f517fbe265b8fe72e24406f95ae811242\": rpc error: code = NotFound desc = could not find container \"3e0055bc1359fbc86f49a9c8d38d174f517fbe265b8fe72e24406f95ae811242\": container with ID starting with 3e0055bc1359fbc86f49a9c8d38d174f517fbe265b8fe72e24406f95ae811242 not found: ID does not exist" Oct 30 19:33:06.442296 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:06.442262 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-6vdd2"] Oct 30 19:33:06.489141 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:06.489105 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-6vdd2"] Oct 30 19:33:06.580411 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:06.580323 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0d5ddf7-16c6-4ec0-933a-e95a8e5b7065" path="/var/lib/kubelet/pods/c0d5ddf7-16c6-4ec0-933a-e95a8e5b7065/volumes" Oct 30 19:33:13.219957 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:33:13.219867 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:33:17.249272 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:17.249229 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dshrx" Oct 30 19:33:39.844382 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:39.844328 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7d7f7cf6c4-bcgck"] Oct 30 19:33:39.844866 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:39.844610 4292 cpu_manager.go:414] "containerMap: removing container" podUID="c0d5ddf7-16c6-4ec0-933a-e95a8e5b7065" containerName="limitador" Oct 30 19:33:39.844866 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:39.844622 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0d5ddf7-16c6-4ec0-933a-e95a8e5b7065" containerName="limitador" Oct 30 19:33:39.844866 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:39.844691 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0d5ddf7-16c6-4ec0-933a-e95a8e5b7065" containerName="limitador" Oct 30 19:33:39.865403 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:39.865372 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d7f7cf6c4-bcgck" Oct 30 19:33:39.908319 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:39.908291 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7d7f7cf6c4-bcgck"] Oct 30 19:33:39.953297 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:39.953268 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 19:33:39.953490 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:39.953320 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 19:33:39.961519 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:39.961498 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-jg4kd\"" Oct 30 19:33:39.961590 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:39.961505 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 19:33:40.012613 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:40.012582 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/488cec0c-d7a1-4adb-a8c9-923c0072b88d-metrics-certs\") pod \"maas-controller-7d7f7cf6c4-bcgck\" (UID: \"488cec0c-d7a1-4adb-a8c9-923c0072b88d\") " pod="opendatahub/maas-controller-7d7f7cf6c4-bcgck" Oct 30 19:33:40.012762 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:40.012632 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/488cec0c-d7a1-4adb-a8c9-923c0072b88d-webhook-cert\") pod \"maas-controller-7d7f7cf6c4-bcgck\" (UID: \"488cec0c-d7a1-4adb-a8c9-923c0072b88d\") " pod="opendatahub/maas-controller-7d7f7cf6c4-bcgck" Oct 30 19:33:40.012762 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:40.012723 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqf6q\" (UniqueName: \"kubernetes.io/projected/488cec0c-d7a1-4adb-a8c9-923c0072b88d-kube-api-access-hqf6q\") pod \"maas-controller-7d7f7cf6c4-bcgck\" (UID: \"488cec0c-d7a1-4adb-a8c9-923c0072b88d\") " pod="opendatahub/maas-controller-7d7f7cf6c4-bcgck" Oct 30 19:33:40.113424 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:40.113311 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/488cec0c-d7a1-4adb-a8c9-923c0072b88d-metrics-certs\") pod \"maas-controller-7d7f7cf6c4-bcgck\" (UID: \"488cec0c-d7a1-4adb-a8c9-923c0072b88d\") " pod="opendatahub/maas-controller-7d7f7cf6c4-bcgck" Oct 30 19:33:40.113590 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:40.113487 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/488cec0c-d7a1-4adb-a8c9-923c0072b88d-webhook-cert\") pod \"maas-controller-7d7f7cf6c4-bcgck\" (UID: \"488cec0c-d7a1-4adb-a8c9-923c0072b88d\") " pod="opendatahub/maas-controller-7d7f7cf6c4-bcgck" Oct 30 19:33:40.113590 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:40.113533 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqf6q\" (UniqueName: \"kubernetes.io/projected/488cec0c-d7a1-4adb-a8c9-923c0072b88d-kube-api-access-hqf6q\") pod \"maas-controller-7d7f7cf6c4-bcgck\" (UID: \"488cec0c-d7a1-4adb-a8c9-923c0072b88d\") " pod="opendatahub/maas-controller-7d7f7cf6c4-bcgck" Oct 30 19:33:40.115763 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:40.115733 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/488cec0c-d7a1-4adb-a8c9-923c0072b88d-metrics-certs\") pod \"maas-controller-7d7f7cf6c4-bcgck\" (UID: \"488cec0c-d7a1-4adb-a8c9-923c0072b88d\") " pod="opendatahub/maas-controller-7d7f7cf6c4-bcgck" Oct 30 19:33:40.115870 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:40.115826 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/488cec0c-d7a1-4adb-a8c9-923c0072b88d-webhook-cert\") pod \"maas-controller-7d7f7cf6c4-bcgck\" (UID: \"488cec0c-d7a1-4adb-a8c9-923c0072b88d\") " pod="opendatahub/maas-controller-7d7f7cf6c4-bcgck" Oct 30 19:33:40.172704 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:40.172664 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqf6q\" (UniqueName: \"kubernetes.io/projected/488cec0c-d7a1-4adb-a8c9-923c0072b88d-kube-api-access-hqf6q\") pod \"maas-controller-7d7f7cf6c4-bcgck\" (UID: \"488cec0c-d7a1-4adb-a8c9-923c0072b88d\") " pod="opendatahub/maas-controller-7d7f7cf6c4-bcgck" Oct 30 19:33:40.178524 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:40.178489 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d7f7cf6c4-bcgck" Oct 30 19:33:40.547557 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:40.547524 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7d7f7cf6c4-bcgck"] Oct 30 19:33:40.554449 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:33:40.554418 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod488cec0c_d7a1_4adb_a8c9_923c0072b88d.slice/crio-1a3e5188db065a5f9442461f770a6c80ffd99c68531e540d812e650fee895573 WatchSource:0}: Error finding container 1a3e5188db065a5f9442461f770a6c80ffd99c68531e540d812e650fee895573: Status 404 returned error can't find the container with id 1a3e5188db065a5f9442461f770a6c80ffd99c68531e540d812e650fee895573 Oct 30 19:33:41.419142 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:41.419099 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d7f7cf6c4-bcgck" event={"ID":"488cec0c-d7a1-4adb-a8c9-923c0072b88d","Type":"ContainerStarted","Data":"1a3e5188db065a5f9442461f770a6c80ffd99c68531e540d812e650fee895573"} Oct 30 19:33:43.225472 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:33:43.225418 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:33:47.445512 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:47.445473 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d7f7cf6c4-bcgck" event={"ID":"488cec0c-d7a1-4adb-a8c9-923c0072b88d","Type":"ContainerStarted","Data":"466a9a514544e8f7bb39c1be6f0fde79e5b04fd8bf3b34528b8088a0ddfe1e02"} Oct 30 19:33:47.445997 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:47.445613 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7d7f7cf6c4-bcgck" Oct 30 19:33:47.569680 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:47.569621 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7d7f7cf6c4-bcgck" podStartSLOduration=2.340927308 podStartE2EDuration="8.569605328s" totalImagesPullingTime="6.22867802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:47.568445487 +0000 UTC m=+531.086760959" watchObservedRunningTime="2026-10-30 19:33:47.569605328 +0000 UTC m=+531.087920837" Oct 30 19:33:50.530682 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:50.530646 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ljppk"] Oct 30 19:33:50.537191 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:50.537167 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ljppk" Oct 30 19:33:50.587645 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:50.587592 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Oct 30 19:33:50.634553 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:50.634516 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Oct 30 19:33:50.634553 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:50.634522 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Oct 30 19:33:50.635844 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:50.635822 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ljppk"] Oct 30 19:33:50.681560 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:50.681533 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-bnks2\"" Oct 30 19:33:50.704786 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:50.704760 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/8f379085-d64c-422c-88e3-b81271212343-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ljppk\" (UID: \"8f379085-d64c-422c-88e3-b81271212343\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ljppk" Oct 30 19:33:50.704934 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:50.704791 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8f379085-d64c-422c-88e3-b81271212343-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ljppk\" (UID: \"8f379085-d64c-422c-88e3-b81271212343\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ljppk" Oct 30 19:33:50.705140 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:50.705103 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f379085-d64c-422c-88e3-b81271212343-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ljppk\" (UID: \"8f379085-d64c-422c-88e3-b81271212343\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ljppk" Oct 30 19:33:50.705224 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:50.705159 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2v9r\" (UniqueName: \"kubernetes.io/projected/8f379085-d64c-422c-88e3-b81271212343-kube-api-access-k2v9r\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ljppk\" (UID: \"8f379085-d64c-422c-88e3-b81271212343\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ljppk" Oct 30 19:33:50.806517 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:50.806424 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/8f379085-d64c-422c-88e3-b81271212343-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ljppk\" (UID: \"8f379085-d64c-422c-88e3-b81271212343\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ljppk" Oct 30 19:33:50.806517 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:50.806470 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8f379085-d64c-422c-88e3-b81271212343-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ljppk\" (UID: \"8f379085-d64c-422c-88e3-b81271212343\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ljppk" Oct 30 19:33:50.806517 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:50.806514 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f379085-d64c-422c-88e3-b81271212343-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ljppk\" (UID: \"8f379085-d64c-422c-88e3-b81271212343\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ljppk" Oct 30 19:33:50.806803 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:50.806532 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2v9r\" (UniqueName: \"kubernetes.io/projected/8f379085-d64c-422c-88e3-b81271212343-kube-api-access-k2v9r\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ljppk\" (UID: \"8f379085-d64c-422c-88e3-b81271212343\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ljppk" Oct 30 19:33:50.806803 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:33:50.806544 4292 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found Oct 30 19:33:50.806803 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:33:50.806618 4292 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8f379085-d64c-422c-88e3-b81271212343-proxy-source podName:8f379085-d64c-422c-88e3-b81271212343 nodeName:}" failed. No retries permitted until 2026-10-30 19:33:51.306597921 +0000 UTC m=+534.824913374 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/8f379085-d64c-422c-88e3-b81271212343-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-ljppk" (UID: "8f379085-d64c-422c-88e3-b81271212343") : configmap "usage-tenancy-proxy-source" not found Oct 30 19:33:50.806803 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:33:50.806663 4292 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found Oct 30 19:33:50.806803 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:33:50.806746 4292 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f379085-d64c-422c-88e3-b81271212343-tls-cert podName:8f379085-d64c-422c-88e3-b81271212343 nodeName:}" failed. No retries permitted until 2026-10-30 19:33:51.306728247 +0000 UTC m=+534.825043698 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/8f379085-d64c-422c-88e3-b81271212343-tls-cert") pod "usage-logs-tenancy-proxy-9c5d49bcd-ljppk" (UID: "8f379085-d64c-422c-88e3-b81271212343") : secret "usage-logs-tenancy-proxy-tls" not found Oct 30 19:33:50.807187 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:50.807167 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f379085-d64c-422c-88e3-b81271212343-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ljppk\" (UID: \"8f379085-d64c-422c-88e3-b81271212343\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ljppk" Oct 30 19:33:50.870403 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:50.870337 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2v9r\" (UniqueName: \"kubernetes.io/projected/8f379085-d64c-422c-88e3-b81271212343-kube-api-access-k2v9r\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ljppk\" (UID: \"8f379085-d64c-422c-88e3-b81271212343\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ljppk" Oct 30 19:33:51.311568 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:51.311528 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/8f379085-d64c-422c-88e3-b81271212343-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ljppk\" (UID: \"8f379085-d64c-422c-88e3-b81271212343\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ljppk" Oct 30 19:33:51.311568 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:51.311568 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8f379085-d64c-422c-88e3-b81271212343-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ljppk\" (UID: \"8f379085-d64c-422c-88e3-b81271212343\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ljppk" Oct 30 19:33:51.312463 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:51.312434 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/8f379085-d64c-422c-88e3-b81271212343-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ljppk\" (UID: \"8f379085-d64c-422c-88e3-b81271212343\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ljppk" Oct 30 19:33:51.314029 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:51.314011 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8f379085-d64c-422c-88e3-b81271212343-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ljppk\" (UID: \"8f379085-d64c-422c-88e3-b81271212343\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ljppk" Oct 30 19:33:51.449346 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:51.449312 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ljppk" Oct 30 19:33:51.818958 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:51.818924 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ljppk"] Oct 30 19:33:51.826093 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:33:51.826068 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f379085_d64c_422c_88e3_b81271212343.slice/crio-0225a8764487d90077f34b6084bcf3c456ffe1d7f874f1d9ec6ec987a4b24965 WatchSource:0}: Error finding container 0225a8764487d90077f34b6084bcf3c456ffe1d7f874f1d9ec6ec987a4b24965: Status 404 returned error can't find the container with id 0225a8764487d90077f34b6084bcf3c456ffe1d7f874f1d9ec6ec987a4b24965 Oct 30 19:33:52.467403 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:52.467345 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ljppk" event={"ID":"8f379085-d64c-422c-88e3-b81271212343","Type":"ContainerStarted","Data":"0225a8764487d90077f34b6084bcf3c456ffe1d7f874f1d9ec6ec987a4b24965"} Oct 30 19:33:54.736291 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:54.736256 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9md6"] Oct 30 19:33:54.741732 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:54.741709 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9md6" Oct 30 19:33:54.838658 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:54.838573 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9md6"] Oct 30 19:33:54.849906 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:54.849873 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/97288e93-eff2-4d48-8cc5-78588579066a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9md6\" (UID: \"97288e93-eff2-4d48-8cc5-78588579066a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9md6" Oct 30 19:33:54.850088 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:54.849935 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97288e93-eff2-4d48-8cc5-78588579066a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9md6\" (UID: \"97288e93-eff2-4d48-8cc5-78588579066a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9md6" Oct 30 19:33:54.850088 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:54.850048 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/97288e93-eff2-4d48-8cc5-78588579066a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9md6\" (UID: \"97288e93-eff2-4d48-8cc5-78588579066a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9md6" Oct 30 19:33:54.850177 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:54.850101 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/97288e93-eff2-4d48-8cc5-78588579066a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9md6\" (UID: \"97288e93-eff2-4d48-8cc5-78588579066a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9md6" Oct 30 19:33:54.850254 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:54.850235 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/97288e93-eff2-4d48-8cc5-78588579066a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9md6\" (UID: \"97288e93-eff2-4d48-8cc5-78588579066a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9md6" Oct 30 19:33:54.850300 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:54.850285 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/97288e93-eff2-4d48-8cc5-78588579066a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9md6\" (UID: \"97288e93-eff2-4d48-8cc5-78588579066a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9md6" Oct 30 19:33:54.850406 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:54.850392 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmxdd\" (UniqueName: \"kubernetes.io/projected/97288e93-eff2-4d48-8cc5-78588579066a-kube-api-access-wmxdd\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9md6\" (UID: \"97288e93-eff2-4d48-8cc5-78588579066a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9md6" Oct 30 19:33:54.950907 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:54.950867 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97288e93-eff2-4d48-8cc5-78588579066a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9md6\" (UID: \"97288e93-eff2-4d48-8cc5-78588579066a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9md6" Oct 30 19:33:54.950907 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:54.950908 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/97288e93-eff2-4d48-8cc5-78588579066a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9md6\" (UID: \"97288e93-eff2-4d48-8cc5-78588579066a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9md6" Oct 30 19:33:54.951160 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:54.950927 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/97288e93-eff2-4d48-8cc5-78588579066a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9md6\" (UID: \"97288e93-eff2-4d48-8cc5-78588579066a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9md6" Oct 30 19:33:54.951160 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:54.950959 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/97288e93-eff2-4d48-8cc5-78588579066a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9md6\" (UID: \"97288e93-eff2-4d48-8cc5-78588579066a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9md6" Oct 30 19:33:54.951160 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:54.950988 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/97288e93-eff2-4d48-8cc5-78588579066a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9md6\" (UID: \"97288e93-eff2-4d48-8cc5-78588579066a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9md6" Oct 30 19:33:54.951160 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:54.951010 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmxdd\" (UniqueName: \"kubernetes.io/projected/97288e93-eff2-4d48-8cc5-78588579066a-kube-api-access-wmxdd\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9md6\" (UID: \"97288e93-eff2-4d48-8cc5-78588579066a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9md6" Oct 30 19:33:54.951160 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:54.951039 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/97288e93-eff2-4d48-8cc5-78588579066a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9md6\" (UID: \"97288e93-eff2-4d48-8cc5-78588579066a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9md6" Oct 30 19:33:54.951750 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:54.951717 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/97288e93-eff2-4d48-8cc5-78588579066a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9md6\" (UID: \"97288e93-eff2-4d48-8cc5-78588579066a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9md6" Oct 30 19:33:54.953434 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:54.953405 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/97288e93-eff2-4d48-8cc5-78588579066a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9md6\" (UID: \"97288e93-eff2-4d48-8cc5-78588579066a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9md6" Oct 30 19:33:54.953531 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:54.953458 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/97288e93-eff2-4d48-8cc5-78588579066a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9md6\" (UID: \"97288e93-eff2-4d48-8cc5-78588579066a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9md6" Oct 30 19:33:54.953798 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:54.953779 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/97288e93-eff2-4d48-8cc5-78588579066a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9md6\" (UID: \"97288e93-eff2-4d48-8cc5-78588579066a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9md6" Oct 30 19:33:54.953928 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:54.953905 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/97288e93-eff2-4d48-8cc5-78588579066a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9md6\" (UID: \"97288e93-eff2-4d48-8cc5-78588579066a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9md6" Oct 30 19:33:55.009216 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:55.009176 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97288e93-eff2-4d48-8cc5-78588579066a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9md6\" (UID: \"97288e93-eff2-4d48-8cc5-78588579066a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9md6" Oct 30 19:33:55.009216 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:55.009199 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmxdd\" (UniqueName: \"kubernetes.io/projected/97288e93-eff2-4d48-8cc5-78588579066a-kube-api-access-wmxdd\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9md6\" (UID: \"97288e93-eff2-4d48-8cc5-78588579066a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9md6" Oct 30 19:33:55.052471 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:55.052435 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9md6" Oct 30 19:33:55.477623 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:55.477544 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9md6"] Oct 30 19:33:55.484320 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:33:55.484293 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97288e93_eff2_4d48_8cc5_78588579066a.slice/crio-65328b106c0798d8257d9554fc95cf5e3b608a903c973c369d6045704a8d2806 WatchSource:0}: Error finding container 65328b106c0798d8257d9554fc95cf5e3b608a903c973c369d6045704a8d2806: Status 404 returned error can't find the container with id 65328b106c0798d8257d9554fc95cf5e3b608a903c973c369d6045704a8d2806 Oct 30 19:33:55.486422 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:55.486332 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:33:55.486522 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:55.486426 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:33:56.487550 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:56.487506 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9md6" event={"ID":"97288e93-eff2-4d48-8cc5-78588579066a","Type":"ContainerStarted","Data":"56f1d9e1c5fef3c2c42aa21b7e339d9e14eb8af07b867ec5121c0aaba25ff47e"} Oct 30 19:33:56.488049 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:56.487563 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9md6" event={"ID":"97288e93-eff2-4d48-8cc5-78588579066a","Type":"ContainerStarted","Data":"65328b106c0798d8257d9554fc95cf5e3b608a903c973c369d6045704a8d2806"} Oct 30 19:33:56.488049 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:56.488044 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9md6" Oct 30 19:33:56.490131 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:56.490091 4292 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-b9md6 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:33:56.490267 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:56.490170 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9md6" podUID="97288e93-eff2-4d48-8cc5-78588579066a" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:33:56.699243 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:56.699174 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9md6" podStartSLOduration=2.699152936 podStartE2EDuration="2.699152936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:56.65229326 +0000 UTC m=+540.170608718" watchObservedRunningTime="2026-10-30 19:33:56.699152936 +0000 UTC m=+540.217468395" Oct 30 19:33:57.493419 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:57.493387 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9md6" Oct 30 19:33:57.712385 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:57.709449 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-qqwkz"] Oct 30 19:33:57.712385 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:57.709681 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qqwkz" podUID="ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9" containerName="discovery" containerID="cri-o://3be37d274e06e8f57658bbfe09b63b2ca7a5dbb4b4bf1478b0db6ee3302e41aa" gracePeriod=30 Oct 30 19:33:57.893434 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:57.893393 4292 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-qqwkz container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.18:8080/ready\": dial tcp 10.133.0.18:8080: connect: connection refused" start-of-body= Oct 30 19:33:57.893608 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:57.893454 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qqwkz" podUID="ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.18:8080/ready\": dial tcp 10.133.0.18:8080: connect: connection refused" Oct 30 19:33:58.089303 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:58.089226 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qqwkz" Oct 30 19:33:58.180002 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:58.179967 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9-istio-csr-ca-configmap\") pod \"ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9\" (UID: \"ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9\") " Oct 30 19:33:58.180188 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:58.180023 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9-local-certs\") pod \"ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9\" (UID: \"ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9\") " Oct 30 19:33:58.180188 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:58.180058 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9-istio-kubeconfig\") pod \"ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9\" (UID: \"ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9\") " Oct 30 19:33:58.180188 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:58.180111 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9-kube-api-access-pcrnj\" (UniqueName: \"kubernetes.io/projected/ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9-kube-api-access-pcrnj\") pod \"ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9\" (UID: \"ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9\") " Oct 30 19:33:58.180378 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:58.180224 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9-istio-csr-dns-cert\") pod \"ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9\" (UID: \"ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9\") " Oct 30 19:33:58.180378 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:58.180321 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9-cacerts\" (UniqueName: \"kubernetes.io/secret/ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9-cacerts\") pod \"ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9\" (UID: \"ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9\") " Oct 30 19:33:58.180495 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:58.180377 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9-istio-token\" (UniqueName: \"kubernetes.io/projected/ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9-istio-token\") pod \"ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9\" (UID: \"ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9\") " Oct 30 19:33:58.180495 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:58.180461 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9-istio-csr-ca-configmap" pod "ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9" (UID: "ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:33:58.180692 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:58.180670 4292 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9-istio-csr-ca-configmap\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:58.182917 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:58.182891 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9-cacerts" pod "ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9" (UID: "ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:33:58.183037 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:58.182987 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9-local-certs" pod "ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9" (UID: "ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:58.183167 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:58.183149 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9-istio-csr-dns-cert" pod "ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9" (UID: "ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:33:58.183237 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:58.183175 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9-kube-api-access-pcrnj" pod "ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9" (UID: "ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9"). InnerVolumeSpecName "kube-api-access-pcrnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:33:58.183745 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:58.183711 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9-istio-kubeconfig" pod "ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9" (UID: "ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:33:58.184871 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:58.184854 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9-istio-token" pod "ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9" (UID: "ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:33:58.281939 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:58.281901 4292 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9-local-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:58.281939 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:58.281931 4292 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9-istio-kubeconfig\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:58.281939 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:58.281943 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcrnj\" (UniqueName: \"kubernetes.io/projected/ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9-kube-api-access-pcrnj\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:58.282198 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:58.281953 4292 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9-istio-csr-dns-cert\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:58.282198 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:58.281962 4292 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9-cacerts\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:58.282198 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:58.281972 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9-istio-token\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:58.452585 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:58.452508 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7d7f7cf6c4-bcgck" Oct 30 19:33:58.498928 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:58.498877 4292 generic.go:356] "Generic (PLEG): container finished" podID="ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9" containerID="3be37d274e06e8f57658bbfe09b63b2ca7a5dbb4b4bf1478b0db6ee3302e41aa" exitCode=0 Oct 30 19:33:58.499411 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:58.498949 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qqwkz" Oct 30 19:33:58.499411 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:58.498965 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qqwkz" event={"ID":"ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9","Type":"ContainerDied","Data":"3be37d274e06e8f57658bbfe09b63b2ca7a5dbb4b4bf1478b0db6ee3302e41aa"} Oct 30 19:33:58.499411 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:58.499035 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qqwkz" event={"ID":"ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9","Type":"ContainerDied","Data":"e8ca3ba2e915507bf16846eb01f0c12feddbafe72e45184f649aba5436ef1245"} Oct 30 19:33:58.499411 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:58.499067 4292 scope.go:122] "RemoveContainer" containerID="3be37d274e06e8f57658bbfe09b63b2ca7a5dbb4b4bf1478b0db6ee3302e41aa" Oct 30 19:33:58.509970 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:58.509945 4292 scope.go:122] "RemoveContainer" containerID="3be37d274e06e8f57658bbfe09b63b2ca7a5dbb4b4bf1478b0db6ee3302e41aa" Oct 30 19:33:58.510280 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:33:58.510256 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3be37d274e06e8f57658bbfe09b63b2ca7a5dbb4b4bf1478b0db6ee3302e41aa\": container with ID starting with 3be37d274e06e8f57658bbfe09b63b2ca7a5dbb4b4bf1478b0db6ee3302e41aa not found: ID does not exist" containerID="3be37d274e06e8f57658bbfe09b63b2ca7a5dbb4b4bf1478b0db6ee3302e41aa" Oct 30 19:33:58.510397 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:58.510288 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3be37d274e06e8f57658bbfe09b63b2ca7a5dbb4b4bf1478b0db6ee3302e41aa"} err="failed to get container status \"3be37d274e06e8f57658bbfe09b63b2ca7a5dbb4b4bf1478b0db6ee3302e41aa\": rpc error: code = NotFound desc = could not find container \"3be37d274e06e8f57658bbfe09b63b2ca7a5dbb4b4bf1478b0db6ee3302e41aa\": container with ID starting with 3be37d274e06e8f57658bbfe09b63b2ca7a5dbb4b4bf1478b0db6ee3302e41aa not found: ID does not exist" Oct 30 19:33:58.676235 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:58.676137 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-qqwkz"] Oct 30 19:33:58.677579 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:33:58.677545 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-qqwkz"] Oct 30 19:34:00.453142 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:00.453109 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-qs7n5"] Oct 30 19:34:00.453681 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:00.453497 4292 cpu_manager.go:414] "containerMap: removing container" podUID="ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9" containerName="discovery" Oct 30 19:34:00.453681 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:00.453514 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9" containerName="discovery" Oct 30 19:34:00.453681 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:00.453597 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9" containerName="discovery" Oct 30 19:34:00.480315 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:00.480284 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qs7n5" Oct 30 19:34:00.503166 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:00.503133 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j44gr\" (UniqueName: \"kubernetes.io/projected/cd1c16db-45d1-4495-a96e-7cbd8d0e5eb8-kube-api-access-j44gr\") pod \"llmisvc-controller-manager-869d5dcc5d-qs7n5\" (UID: \"cd1c16db-45d1-4495-a96e-7cbd8d0e5eb8\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qs7n5" Oct 30 19:34:00.503303 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:00.503213 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd1c16db-45d1-4495-a96e-7cbd8d0e5eb8-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-qs7n5\" (UID: \"cd1c16db-45d1-4495-a96e-7cbd8d0e5eb8\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qs7n5" Oct 30 19:34:00.511617 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:00.511579 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-qs7n5"] Oct 30 19:34:00.531551 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:00.531507 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 30 19:34:00.531730 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:00.531577 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-4bcvr\"" Oct 30 19:34:00.580677 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:00.580647 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9" path="/var/lib/kubelet/pods/ac1a2e35-c21e-42b4-b8d2-fc0b7e7cadf9/volumes" Oct 30 19:34:00.604521 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:00.604484 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j44gr\" (UniqueName: \"kubernetes.io/projected/cd1c16db-45d1-4495-a96e-7cbd8d0e5eb8-kube-api-access-j44gr\") pod \"llmisvc-controller-manager-869d5dcc5d-qs7n5\" (UID: \"cd1c16db-45d1-4495-a96e-7cbd8d0e5eb8\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qs7n5" Oct 30 19:34:00.604679 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:00.604548 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd1c16db-45d1-4495-a96e-7cbd8d0e5eb8-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-qs7n5\" (UID: \"cd1c16db-45d1-4495-a96e-7cbd8d0e5eb8\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qs7n5" Oct 30 19:34:00.607042 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:00.607003 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd1c16db-45d1-4495-a96e-7cbd8d0e5eb8-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-qs7n5\" (UID: \"cd1c16db-45d1-4495-a96e-7cbd8d0e5eb8\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qs7n5" Oct 30 19:34:00.672042 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:00.672004 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j44gr\" (UniqueName: \"kubernetes.io/projected/cd1c16db-45d1-4495-a96e-7cbd8d0e5eb8-kube-api-access-j44gr\") pod \"llmisvc-controller-manager-869d5dcc5d-qs7n5\" (UID: \"cd1c16db-45d1-4495-a96e-7cbd8d0e5eb8\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qs7n5" Oct 30 19:34:00.796018 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:00.795983 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qs7n5" Oct 30 19:34:01.191885 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:01.191803 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-qs7n5"] Oct 30 19:34:01.198639 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:34:01.198611 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podcd1c16db_45d1_4495_a96e_7cbd8d0e5eb8.slice/crio-40a85cd4cd6d8522b51724422f1b7cb5b4bdb84bfe72b353d73de9dd6f05d968 WatchSource:0}: Error finding container 40a85cd4cd6d8522b51724422f1b7cb5b4bdb84bfe72b353d73de9dd6f05d968: Status 404 returned error can't find the container with id 40a85cd4cd6d8522b51724422f1b7cb5b4bdb84bfe72b353d73de9dd6f05d968 Oct 30 19:34:01.515597 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:01.515504 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qs7n5" event={"ID":"cd1c16db-45d1-4495-a96e-7cbd8d0e5eb8","Type":"ContainerStarted","Data":"40a85cd4cd6d8522b51724422f1b7cb5b4bdb84bfe72b353d73de9dd6f05d968"} Oct 30 19:34:04.538502 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:04.538451 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qs7n5" event={"ID":"cd1c16db-45d1-4495-a96e-7cbd8d0e5eb8","Type":"ContainerStarted","Data":"9f9201b3aefff8a940a6a9d6ebcf7f41c8b4d1876b30ac06e1e4bda8ceabf70d"} Oct 30 19:34:04.538977 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:04.538643 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qs7n5" Oct 30 19:34:04.649793 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:04.649737 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qs7n5" podStartSLOduration=1.848207535 podStartE2EDuration="4.649725158s" totalImagesPullingTime="2.801517623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:04.649437716 +0000 UTC m=+548.167753174" watchObservedRunningTime="2026-10-30 19:34:04.649725158 +0000 UTC m=+548.168040658" Oct 30 19:34:09.560475 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:09.560436 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ljppk" event={"ID":"8f379085-d64c-422c-88e3-b81271212343","Type":"ContainerStarted","Data":"5841b1620c6f5471ba31648143f12b6f2cea16527c3426fd00f3b33457bf68a5"} Oct 30 19:34:09.560902 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:09.560519 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ljppk" Oct 30 19:34:09.671845 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:09.671788 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ljppk" podStartSLOduration=2.524263839 podStartE2EDuration="19.67177502s" totalImagesPullingTime="17.147511181s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:09.671280059 +0000 UTC m=+553.189595538" watchObservedRunningTime="2026-10-30 19:34:09.67177502 +0000 UTC m=+553.190090477" Oct 30 19:34:13.220477 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:34:13.220436 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:34:20.570542 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:20.570511 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ljppk" Oct 30 19:34:23.950368 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:23.950320 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-597fb6789c-vz7w2"] Oct 30 19:34:23.953736 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:23.953715 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-597fb6789c-vz7w2" Oct 30 19:34:24.003781 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:24.003741 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:34:24.050397 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:24.050362 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 19:34:24.050397 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:24.050371 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:34:24.050642 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:24.050502 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 19:34:24.050642 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:24.050566 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-skrr4\"" Oct 30 19:34:24.051964 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:24.051947 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-597fb6789c-vz7w2"] Oct 30 19:34:24.106586 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:24.106549 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/051a9d46-ac94-4919-901d-e230d6113fc7-maas-api-tls\") pod \"maas-api-597fb6789c-vz7w2\" (UID: \"051a9d46-ac94-4919-901d-e230d6113fc7\") " pod="odh-ai-gateway-infra/maas-api-597fb6789c-vz7w2" Oct 30 19:34:24.106763 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:24.106595 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/051a9d46-ac94-4919-901d-e230d6113fc7-metrics-certs\") pod \"maas-api-597fb6789c-vz7w2\" (UID: \"051a9d46-ac94-4919-901d-e230d6113fc7\") " pod="odh-ai-gateway-infra/maas-api-597fb6789c-vz7w2" Oct 30 19:34:24.106763 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:24.106632 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/051a9d46-ac94-4919-901d-e230d6113fc7-openshift-service-ca\") pod \"maas-api-597fb6789c-vz7w2\" (UID: \"051a9d46-ac94-4919-901d-e230d6113fc7\") " pod="odh-ai-gateway-infra/maas-api-597fb6789c-vz7w2" Oct 30 19:34:24.106763 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:24.106675 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hj2f\" (UniqueName: \"kubernetes.io/projected/051a9d46-ac94-4919-901d-e230d6113fc7-kube-api-access-6hj2f\") pod \"maas-api-597fb6789c-vz7w2\" (UID: \"051a9d46-ac94-4919-901d-e230d6113fc7\") " pod="odh-ai-gateway-infra/maas-api-597fb6789c-vz7w2" Oct 30 19:34:24.207478 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:24.207393 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/051a9d46-ac94-4919-901d-e230d6113fc7-openshift-service-ca\") pod \"maas-api-597fb6789c-vz7w2\" (UID: \"051a9d46-ac94-4919-901d-e230d6113fc7\") " pod="odh-ai-gateway-infra/maas-api-597fb6789c-vz7w2" Oct 30 19:34:24.207478 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:24.207427 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hj2f\" (UniqueName: \"kubernetes.io/projected/051a9d46-ac94-4919-901d-e230d6113fc7-kube-api-access-6hj2f\") pod \"maas-api-597fb6789c-vz7w2\" (UID: \"051a9d46-ac94-4919-901d-e230d6113fc7\") " pod="odh-ai-gateway-infra/maas-api-597fb6789c-vz7w2" Oct 30 19:34:24.207710 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:24.207493 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/051a9d46-ac94-4919-901d-e230d6113fc7-maas-api-tls\") pod \"maas-api-597fb6789c-vz7w2\" (UID: \"051a9d46-ac94-4919-901d-e230d6113fc7\") " pod="odh-ai-gateway-infra/maas-api-597fb6789c-vz7w2" Oct 30 19:34:24.207710 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:24.207520 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/051a9d46-ac94-4919-901d-e230d6113fc7-metrics-certs\") pod \"maas-api-597fb6789c-vz7w2\" (UID: \"051a9d46-ac94-4919-901d-e230d6113fc7\") " pod="odh-ai-gateway-infra/maas-api-597fb6789c-vz7w2" Oct 30 19:34:24.208108 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:24.208085 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/051a9d46-ac94-4919-901d-e230d6113fc7-openshift-service-ca\") pod \"maas-api-597fb6789c-vz7w2\" (UID: \"051a9d46-ac94-4919-901d-e230d6113fc7\") " pod="odh-ai-gateway-infra/maas-api-597fb6789c-vz7w2" Oct 30 19:34:24.209989 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:24.209967 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/051a9d46-ac94-4919-901d-e230d6113fc7-metrics-certs\") pod \"maas-api-597fb6789c-vz7w2\" (UID: \"051a9d46-ac94-4919-901d-e230d6113fc7\") " pod="odh-ai-gateway-infra/maas-api-597fb6789c-vz7w2" Oct 30 19:34:24.210088 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:24.210023 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/051a9d46-ac94-4919-901d-e230d6113fc7-maas-api-tls\") pod \"maas-api-597fb6789c-vz7w2\" (UID: \"051a9d46-ac94-4919-901d-e230d6113fc7\") " pod="odh-ai-gateway-infra/maas-api-597fb6789c-vz7w2" Oct 30 19:34:24.265604 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:24.265567 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hj2f\" (UniqueName: \"kubernetes.io/projected/051a9d46-ac94-4919-901d-e230d6113fc7-kube-api-access-6hj2f\") pod \"maas-api-597fb6789c-vz7w2\" (UID: \"051a9d46-ac94-4919-901d-e230d6113fc7\") " pod="odh-ai-gateway-infra/maas-api-597fb6789c-vz7w2" Oct 30 19:34:24.265783 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:24.265705 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-597fb6789c-vz7w2" Oct 30 19:34:24.634156 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:24.634124 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-597fb6789c-vz7w2"] Oct 30 19:34:24.641192 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:34:24.641163 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod051a9d46_ac94_4919_901d_e230d6113fc7.slice/crio-e9ef4f2183e0e4fa13837bac6c799e006cb69388ad641d700c9d75ea145e5f44 WatchSource:0}: Error finding container e9ef4f2183e0e4fa13837bac6c799e006cb69388ad641d700c9d75ea145e5f44: Status 404 returned error can't find the container with id e9ef4f2183e0e4fa13837bac6c799e006cb69388ad641d700c9d75ea145e5f44 Oct 30 19:34:25.633493 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:25.633454 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-597fb6789c-vz7w2" event={"ID":"051a9d46-ac94-4919-901d-e230d6113fc7","Type":"ContainerStarted","Data":"e9ef4f2183e0e4fa13837bac6c799e006cb69388ad641d700c9d75ea145e5f44"} Oct 30 19:34:31.658372 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:31.658315 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-597fb6789c-vz7w2" event={"ID":"051a9d46-ac94-4919-901d-e230d6113fc7","Type":"ContainerStarted","Data":"5d253777b921c0254ff038581f9291ddb32435b1fc4510f9d1571b98cafa4514"} Oct 30 19:34:31.658857 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:31.658430 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-597fb6789c-vz7w2" Oct 30 19:34:31.772413 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:31.772310 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-597fb6789c-vz7w2" podStartSLOduration=1.8934617459999998 podStartE2EDuration="8.772296495s" totalImagesPullingTime="6.878834749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:31.771417671 +0000 UTC m=+575.289733125" watchObservedRunningTime="2026-10-30 19:34:31.772296495 +0000 UTC m=+575.290611951" Oct 30 19:34:35.544970 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:35.544936 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qs7n5" Oct 30 19:34:37.666856 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:37.666825 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-597fb6789c-vz7w2" Oct 30 19:34:43.219831 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:34:43.219703 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:34:49.018936 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:49.018899 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-597fb6789c-vz7w2"] Oct 30 19:34:49.019393 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:49.019092 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-597fb6789c-vz7w2" podUID="051a9d46-ac94-4919-901d-e230d6113fc7" containerName="maas-api" containerID="cri-o://5d253777b921c0254ff038581f9291ddb32435b1fc4510f9d1571b98cafa4514" gracePeriod=30 Oct 30 19:34:49.259209 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:49.259184 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-597fb6789c-vz7w2" Oct 30 19:34:49.320306 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:49.320226 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/051a9d46-ac94-4919-901d-e230d6113fc7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/051a9d46-ac94-4919-901d-e230d6113fc7-openshift-service-ca\") pod \"051a9d46-ac94-4919-901d-e230d6113fc7\" (UID: \"051a9d46-ac94-4919-901d-e230d6113fc7\") " Oct 30 19:34:49.320306 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:49.320275 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/051a9d46-ac94-4919-901d-e230d6113fc7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/051a9d46-ac94-4919-901d-e230d6113fc7-maas-api-tls\") pod \"051a9d46-ac94-4919-901d-e230d6113fc7\" (UID: \"051a9d46-ac94-4919-901d-e230d6113fc7\") " Oct 30 19:34:49.320574 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:49.320315 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/051a9d46-ac94-4919-901d-e230d6113fc7-kube-api-access-6hj2f\" (UniqueName: \"kubernetes.io/projected/051a9d46-ac94-4919-901d-e230d6113fc7-kube-api-access-6hj2f\") pod \"051a9d46-ac94-4919-901d-e230d6113fc7\" (UID: \"051a9d46-ac94-4919-901d-e230d6113fc7\") " Oct 30 19:34:49.320574 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:49.320374 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/051a9d46-ac94-4919-901d-e230d6113fc7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/051a9d46-ac94-4919-901d-e230d6113fc7-metrics-certs\") pod \"051a9d46-ac94-4919-901d-e230d6113fc7\" (UID: \"051a9d46-ac94-4919-901d-e230d6113fc7\") " Oct 30 19:34:49.320683 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:49.320627 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/051a9d46-ac94-4919-901d-e230d6113fc7-openshift-service-ca" pod "051a9d46-ac94-4919-901d-e230d6113fc7" (UID: "051a9d46-ac94-4919-901d-e230d6113fc7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:34:49.322654 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:49.322623 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/051a9d46-ac94-4919-901d-e230d6113fc7-metrics-certs" pod "051a9d46-ac94-4919-901d-e230d6113fc7" (UID: "051a9d46-ac94-4919-901d-e230d6113fc7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:34:49.322775 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:49.322672 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/051a9d46-ac94-4919-901d-e230d6113fc7-kube-api-access-6hj2f" pod "051a9d46-ac94-4919-901d-e230d6113fc7" (UID: "051a9d46-ac94-4919-901d-e230d6113fc7"). InnerVolumeSpecName "kube-api-access-6hj2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:34:49.322775 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:49.322673 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/051a9d46-ac94-4919-901d-e230d6113fc7-maas-api-tls" pod "051a9d46-ac94-4919-901d-e230d6113fc7" (UID: "051a9d46-ac94-4919-901d-e230d6113fc7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:34:49.421564 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:49.421530 4292 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/051a9d46-ac94-4919-901d-e230d6113fc7-metrics-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:49.421564 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:49.421558 4292 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/051a9d46-ac94-4919-901d-e230d6113fc7-openshift-service-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:49.421564 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:49.421569 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/051a9d46-ac94-4919-901d-e230d6113fc7-maas-api-tls\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:49.421794 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:49.421579 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hj2f\" (UniqueName: \"kubernetes.io/projected/051a9d46-ac94-4919-901d-e230d6113fc7-kube-api-access-6hj2f\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:49.729630 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:49.729549 4292 generic.go:356] "Generic (PLEG): container finished" podID="051a9d46-ac94-4919-901d-e230d6113fc7" containerID="5d253777b921c0254ff038581f9291ddb32435b1fc4510f9d1571b98cafa4514" exitCode=0 Oct 30 19:34:49.729630 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:49.729595 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-597fb6789c-vz7w2" event={"ID":"051a9d46-ac94-4919-901d-e230d6113fc7","Type":"ContainerDied","Data":"5d253777b921c0254ff038581f9291ddb32435b1fc4510f9d1571b98cafa4514"} Oct 30 19:34:49.729828 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:49.729633 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-597fb6789c-vz7w2" event={"ID":"051a9d46-ac94-4919-901d-e230d6113fc7","Type":"ContainerDied","Data":"e9ef4f2183e0e4fa13837bac6c799e006cb69388ad641d700c9d75ea145e5f44"} Oct 30 19:34:49.729828 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:49.729655 4292 scope.go:122] "RemoveContainer" containerID="5d253777b921c0254ff038581f9291ddb32435b1fc4510f9d1571b98cafa4514" Oct 30 19:34:49.729828 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:49.729597 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-597fb6789c-vz7w2" Oct 30 19:34:49.739721 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:49.739694 4292 scope.go:122] "RemoveContainer" containerID="5d253777b921c0254ff038581f9291ddb32435b1fc4510f9d1571b98cafa4514" Oct 30 19:34:49.740037 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:34:49.740019 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d253777b921c0254ff038581f9291ddb32435b1fc4510f9d1571b98cafa4514\": container with ID starting with 5d253777b921c0254ff038581f9291ddb32435b1fc4510f9d1571b98cafa4514 not found: ID does not exist" containerID="5d253777b921c0254ff038581f9291ddb32435b1fc4510f9d1571b98cafa4514" Oct 30 19:34:49.740080 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:49.740051 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d253777b921c0254ff038581f9291ddb32435b1fc4510f9d1571b98cafa4514"} err="failed to get container status \"5d253777b921c0254ff038581f9291ddb32435b1fc4510f9d1571b98cafa4514\": rpc error: code = NotFound desc = could not find container \"5d253777b921c0254ff038581f9291ddb32435b1fc4510f9d1571b98cafa4514\": container with ID starting with 5d253777b921c0254ff038581f9291ddb32435b1fc4510f9d1571b98cafa4514 not found: ID does not exist" Oct 30 19:34:49.886289 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:49.886248 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-597fb6789c-vz7w2"] Oct 30 19:34:49.890812 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:49.890781 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-597fb6789c-vz7w2"] Oct 30 19:34:50.580185 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:34:50.580149 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="051a9d46-ac94-4919-901d-e230d6113fc7" path="/var/lib/kubelet/pods/051a9d46-ac94-4919-901d-e230d6113fc7/volumes" Oct 30 19:35:13.219304 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:35:13.219268 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:35:23.129915 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:35:23.129875 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-5hwsq"] Oct 30 19:35:23.130642 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:35:23.130200 4292 cpu_manager.go:414] "containerMap: removing container" podUID="051a9d46-ac94-4919-901d-e230d6113fc7" containerName="maas-api" Oct 30 19:35:23.130642 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:35:23.130211 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="051a9d46-ac94-4919-901d-e230d6113fc7" containerName="maas-api" Oct 30 19:35:23.130642 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:35:23.130262 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="051a9d46-ac94-4919-901d-e230d6113fc7" containerName="maas-api" Oct 30 19:35:23.133128 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:35:23.133106 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-5hwsq" Oct 30 19:35:23.189274 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:35:23.189240 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-5hwsq"] Oct 30 19:35:23.207659 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:35:23.207621 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/371c7b75-70f3-4eb8-8ae0-dca9232838cb-cert\") pod \"llmisvc-controller-manager-6466c554d8-5hwsq\" (UID: \"371c7b75-70f3-4eb8-8ae0-dca9232838cb\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-5hwsq" Oct 30 19:35:23.207659 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:35:23.207658 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6x2p\" (UniqueName: \"kubernetes.io/projected/371c7b75-70f3-4eb8-8ae0-dca9232838cb-kube-api-access-q6x2p\") pod \"llmisvc-controller-manager-6466c554d8-5hwsq\" (UID: \"371c7b75-70f3-4eb8-8ae0-dca9232838cb\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-5hwsq" Oct 30 19:35:23.308401 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:35:23.308343 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/371c7b75-70f3-4eb8-8ae0-dca9232838cb-cert\") pod \"llmisvc-controller-manager-6466c554d8-5hwsq\" (UID: \"371c7b75-70f3-4eb8-8ae0-dca9232838cb\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-5hwsq" Oct 30 19:35:23.308401 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:35:23.308412 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6x2p\" (UniqueName: \"kubernetes.io/projected/371c7b75-70f3-4eb8-8ae0-dca9232838cb-kube-api-access-q6x2p\") pod \"llmisvc-controller-manager-6466c554d8-5hwsq\" (UID: \"371c7b75-70f3-4eb8-8ae0-dca9232838cb\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-5hwsq" Oct 30 19:35:23.310859 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:35:23.310837 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/371c7b75-70f3-4eb8-8ae0-dca9232838cb-cert\") pod \"llmisvc-controller-manager-6466c554d8-5hwsq\" (UID: \"371c7b75-70f3-4eb8-8ae0-dca9232838cb\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-5hwsq" Oct 30 19:35:23.373162 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:35:23.373132 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6x2p\" (UniqueName: \"kubernetes.io/projected/371c7b75-70f3-4eb8-8ae0-dca9232838cb-kube-api-access-q6x2p\") pod \"llmisvc-controller-manager-6466c554d8-5hwsq\" (UID: \"371c7b75-70f3-4eb8-8ae0-dca9232838cb\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-5hwsq" Oct 30 19:35:23.445225 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:35:23.445140 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-5hwsq" Oct 30 19:35:23.814709 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:35:23.814671 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-5hwsq"] Oct 30 19:35:23.821804 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:35:23.821763 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod371c7b75_70f3_4eb8_8ae0_dca9232838cb.slice/crio-76578947c49ebc305762ca942096acf05c0460d2bbcd9a956905c522961c8ab7 WatchSource:0}: Error finding container 76578947c49ebc305762ca942096acf05c0460d2bbcd9a956905c522961c8ab7: Status 404 returned error can't find the container with id 76578947c49ebc305762ca942096acf05c0460d2bbcd9a956905c522961c8ab7 Oct 30 19:35:23.865415 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:35:23.865380 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-5hwsq" event={"ID":"371c7b75-70f3-4eb8-8ae0-dca9232838cb","Type":"ContainerStarted","Data":"76578947c49ebc305762ca942096acf05c0460d2bbcd9a956905c522961c8ab7"} Oct 30 19:35:24.870984 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:35:24.870893 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-5hwsq" event={"ID":"371c7b75-70f3-4eb8-8ae0-dca9232838cb","Type":"ContainerStarted","Data":"6f8defbe13c7e7368a522cf71dca1a033c4a64794d11b55eace1c14e5a7ca0a1"} Oct 30 19:35:24.871387 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:35:24.871001 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-5hwsq" Oct 30 19:35:24.982814 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:35:24.982756 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-5hwsq" podStartSLOduration=1.222412971 podStartE2EDuration="1.982731742s" totalImagesPullingTime="760.318771ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:24.982157596 +0000 UTC m=+628.500473053" watchObservedRunningTime="2026-10-30 19:35:24.982731742 +0000 UTC m=+628.501047198" Oct 30 19:35:43.220227 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:35:43.220186 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:35:55.876692 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:35:55.876663 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-5hwsq" Oct 30 19:35:56.036599 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:35:56.036563 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-qs7n5"] Oct 30 19:35:56.036847 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:35:56.036801 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qs7n5" podUID="cd1c16db-45d1-4495-a96e-7cbd8d0e5eb8" containerName="manager" containerID="cri-o://9f9201b3aefff8a940a6a9d6ebcf7f41c8b4d1876b30ac06e1e4bda8ceabf70d" gracePeriod=30 Oct 30 19:35:56.286751 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:35:56.286725 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qs7n5" Oct 30 19:35:56.371260 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:35:56.371223 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd1c16db-45d1-4495-a96e-7cbd8d0e5eb8-cert\" (UniqueName: \"kubernetes.io/secret/cd1c16db-45d1-4495-a96e-7cbd8d0e5eb8-cert\") pod \"cd1c16db-45d1-4495-a96e-7cbd8d0e5eb8\" (UID: \"cd1c16db-45d1-4495-a96e-7cbd8d0e5eb8\") " Oct 30 19:35:56.371494 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:35:56.371400 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd1c16db-45d1-4495-a96e-7cbd8d0e5eb8-kube-api-access-j44gr\" (UniqueName: \"kubernetes.io/projected/cd1c16db-45d1-4495-a96e-7cbd8d0e5eb8-kube-api-access-j44gr\") pod \"cd1c16db-45d1-4495-a96e-7cbd8d0e5eb8\" (UID: \"cd1c16db-45d1-4495-a96e-7cbd8d0e5eb8\") " Oct 30 19:35:56.373495 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:35:56.373465 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd1c16db-45d1-4495-a96e-7cbd8d0e5eb8-cert" pod "cd1c16db-45d1-4495-a96e-7cbd8d0e5eb8" (UID: "cd1c16db-45d1-4495-a96e-7cbd8d0e5eb8"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:35:56.373613 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:35:56.373508 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd1c16db-45d1-4495-a96e-7cbd8d0e5eb8-kube-api-access-j44gr" pod "cd1c16db-45d1-4495-a96e-7cbd8d0e5eb8" (UID: "cd1c16db-45d1-4495-a96e-7cbd8d0e5eb8"). InnerVolumeSpecName "kube-api-access-j44gr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:35:56.472196 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:35:56.472105 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j44gr\" (UniqueName: \"kubernetes.io/projected/cd1c16db-45d1-4495-a96e-7cbd8d0e5eb8-kube-api-access-j44gr\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:56.472196 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:35:56.472136 4292 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd1c16db-45d1-4495-a96e-7cbd8d0e5eb8-cert\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:56.994794 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:35:56.994756 4292 generic.go:356] "Generic (PLEG): container finished" podID="cd1c16db-45d1-4495-a96e-7cbd8d0e5eb8" containerID="9f9201b3aefff8a940a6a9d6ebcf7f41c8b4d1876b30ac06e1e4bda8ceabf70d" exitCode=0 Oct 30 19:35:56.994794 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:35:56.994796 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qs7n5" Oct 30 19:35:56.995301 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:35:56.994816 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qs7n5" event={"ID":"cd1c16db-45d1-4495-a96e-7cbd8d0e5eb8","Type":"ContainerDied","Data":"9f9201b3aefff8a940a6a9d6ebcf7f41c8b4d1876b30ac06e1e4bda8ceabf70d"} Oct 30 19:35:56.995301 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:35:56.994868 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qs7n5" event={"ID":"cd1c16db-45d1-4495-a96e-7cbd8d0e5eb8","Type":"ContainerDied","Data":"40a85cd4cd6d8522b51724422f1b7cb5b4bdb84bfe72b353d73de9dd6f05d968"} Oct 30 19:35:56.995301 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:35:56.994891 4292 scope.go:122] "RemoveContainer" containerID="9f9201b3aefff8a940a6a9d6ebcf7f41c8b4d1876b30ac06e1e4bda8ceabf70d" Oct 30 19:35:57.002769 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:35:57.002754 4292 scope.go:122] "RemoveContainer" containerID="9f9201b3aefff8a940a6a9d6ebcf7f41c8b4d1876b30ac06e1e4bda8ceabf70d" Oct 30 19:35:57.003018 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:35:57.003001 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f9201b3aefff8a940a6a9d6ebcf7f41c8b4d1876b30ac06e1e4bda8ceabf70d\": container with ID starting with 9f9201b3aefff8a940a6a9d6ebcf7f41c8b4d1876b30ac06e1e4bda8ceabf70d not found: ID does not exist" containerID="9f9201b3aefff8a940a6a9d6ebcf7f41c8b4d1876b30ac06e1e4bda8ceabf70d" Oct 30 19:35:57.003061 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:35:57.003027 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f9201b3aefff8a940a6a9d6ebcf7f41c8b4d1876b30ac06e1e4bda8ceabf70d"} err="failed to get container status \"9f9201b3aefff8a940a6a9d6ebcf7f41c8b4d1876b30ac06e1e4bda8ceabf70d\": rpc error: code = NotFound desc = could not find container \"9f9201b3aefff8a940a6a9d6ebcf7f41c8b4d1876b30ac06e1e4bda8ceabf70d\": container with ID starting with 9f9201b3aefff8a940a6a9d6ebcf7f41c8b4d1876b30ac06e1e4bda8ceabf70d not found: ID does not exist" Oct 30 19:35:57.150643 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:35:57.150607 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-qs7n5"] Oct 30 19:35:57.197415 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:35:57.197380 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-qs7n5"] Oct 30 19:35:58.581420 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:35:58.581388 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd1c16db-45d1-4495-a96e-7cbd8d0e5eb8" path="/var/lib/kubelet/pods/cd1c16db-45d1-4495-a96e-7cbd8d0e5eb8/volumes" Oct 30 19:36:13.219513 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:36:13.219433 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:36:43.220871 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:36:43.220831 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:37:00.979482 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:37:00.979446 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kr46d"] Oct 30 19:37:00.979962 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:37:00.979738 4292 cpu_manager.go:414] "containerMap: removing container" podUID="cd1c16db-45d1-4495-a96e-7cbd8d0e5eb8" containerName="manager" Oct 30 19:37:00.979962 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:37:00.979748 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd1c16db-45d1-4495-a96e-7cbd8d0e5eb8" containerName="manager" Oct 30 19:37:00.979962 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:37:00.979819 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd1c16db-45d1-4495-a96e-7cbd8d0e5eb8" containerName="manager" Oct 30 19:37:00.983855 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:37:00.983835 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kr46d" Oct 30 19:37:01.016614 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:37:01.016569 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de3476ef-83a0-4ca2-9b76-227ee4b35d7c-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-kr46d\" (UID: \"de3476ef-83a0-4ca2-9b76-227ee4b35d7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kr46d" Oct 30 19:37:01.016614 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:37:01.016608 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de3476ef-83a0-4ca2-9b76-227ee4b35d7c-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-kr46d\" (UID: \"de3476ef-83a0-4ca2-9b76-227ee4b35d7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kr46d" Oct 30 19:37:01.016882 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:37:01.016644 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79l4d\" (UniqueName: \"kubernetes.io/projected/de3476ef-83a0-4ca2-9b76-227ee4b35d7c-kube-api-access-79l4d\") pod \"maas-default-gateway-openshift-default-694db55c87-kr46d\" (UID: \"de3476ef-83a0-4ca2-9b76-227ee4b35d7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kr46d" Oct 30 19:37:01.016882 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:37:01.016774 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de3476ef-83a0-4ca2-9b76-227ee4b35d7c-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-kr46d\" (UID: \"de3476ef-83a0-4ca2-9b76-227ee4b35d7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kr46d" Oct 30 19:37:01.016882 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:37:01.016820 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de3476ef-83a0-4ca2-9b76-227ee4b35d7c-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-kr46d\" (UID: \"de3476ef-83a0-4ca2-9b76-227ee4b35d7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kr46d" Oct 30 19:37:01.016882 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:37:01.016852 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de3476ef-83a0-4ca2-9b76-227ee4b35d7c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-kr46d\" (UID: \"de3476ef-83a0-4ca2-9b76-227ee4b35d7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kr46d" Oct 30 19:37:01.017021 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:37:01.016969 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de3476ef-83a0-4ca2-9b76-227ee4b35d7c-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-kr46d\" (UID: \"de3476ef-83a0-4ca2-9b76-227ee4b35d7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kr46d" Oct 30 19:37:01.017054 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:37:01.017024 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de3476ef-83a0-4ca2-9b76-227ee4b35d7c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-kr46d\" (UID: \"de3476ef-83a0-4ca2-9b76-227ee4b35d7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kr46d" Oct 30 19:37:01.017054 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:37:01.017045 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de3476ef-83a0-4ca2-9b76-227ee4b35d7c-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-kr46d\" (UID: \"de3476ef-83a0-4ca2-9b76-227ee4b35d7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kr46d" Oct 30 19:37:01.081596 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:37:01.081559 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kr46d"] Oct 30 19:37:01.118275 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:37:01.118214 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de3476ef-83a0-4ca2-9b76-227ee4b35d7c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-kr46d\" (UID: \"de3476ef-83a0-4ca2-9b76-227ee4b35d7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kr46d" Oct 30 19:37:01.118494 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:37:01.118318 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de3476ef-83a0-4ca2-9b76-227ee4b35d7c-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-kr46d\" (UID: \"de3476ef-83a0-4ca2-9b76-227ee4b35d7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kr46d" Oct 30 19:37:01.118494 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:37:01.118386 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de3476ef-83a0-4ca2-9b76-227ee4b35d7c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-kr46d\" (UID: \"de3476ef-83a0-4ca2-9b76-227ee4b35d7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kr46d" Oct 30 19:37:01.118494 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:37:01.118411 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de3476ef-83a0-4ca2-9b76-227ee4b35d7c-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-kr46d\" (UID: \"de3476ef-83a0-4ca2-9b76-227ee4b35d7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kr46d" Oct 30 19:37:01.118494 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:37:01.118458 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de3476ef-83a0-4ca2-9b76-227ee4b35d7c-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-kr46d\" (UID: \"de3476ef-83a0-4ca2-9b76-227ee4b35d7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kr46d" Oct 30 19:37:01.118674 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:37:01.118606 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de3476ef-83a0-4ca2-9b76-227ee4b35d7c-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-kr46d\" (UID: \"de3476ef-83a0-4ca2-9b76-227ee4b35d7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kr46d" Oct 30 19:37:01.118674 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:37:01.118658 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79l4d\" (UniqueName: \"kubernetes.io/projected/de3476ef-83a0-4ca2-9b76-227ee4b35d7c-kube-api-access-79l4d\") pod \"maas-default-gateway-openshift-default-694db55c87-kr46d\" (UID: \"de3476ef-83a0-4ca2-9b76-227ee4b35d7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kr46d" Oct 30 19:37:01.118792 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:37:01.118703 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de3476ef-83a0-4ca2-9b76-227ee4b35d7c-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-kr46d\" (UID: \"de3476ef-83a0-4ca2-9b76-227ee4b35d7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kr46d" Oct 30 19:37:01.118792 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:37:01.118739 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de3476ef-83a0-4ca2-9b76-227ee4b35d7c-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-kr46d\" (UID: \"de3476ef-83a0-4ca2-9b76-227ee4b35d7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kr46d" Oct 30 19:37:01.118898 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:37:01.118847 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de3476ef-83a0-4ca2-9b76-227ee4b35d7c-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-kr46d\" (UID: \"de3476ef-83a0-4ca2-9b76-227ee4b35d7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kr46d" Oct 30 19:37:01.118898 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:37:01.118881 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de3476ef-83a0-4ca2-9b76-227ee4b35d7c-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-kr46d\" (UID: \"de3476ef-83a0-4ca2-9b76-227ee4b35d7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kr46d" Oct 30 19:37:01.119005 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:37:01.118987 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de3476ef-83a0-4ca2-9b76-227ee4b35d7c-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-kr46d\" (UID: \"de3476ef-83a0-4ca2-9b76-227ee4b35d7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kr46d" Oct 30 19:37:01.119207 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:37:01.119180 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de3476ef-83a0-4ca2-9b76-227ee4b35d7c-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-kr46d\" (UID: \"de3476ef-83a0-4ca2-9b76-227ee4b35d7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kr46d" Oct 30 19:37:01.119492 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:37:01.119469 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de3476ef-83a0-4ca2-9b76-227ee4b35d7c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-kr46d\" (UID: \"de3476ef-83a0-4ca2-9b76-227ee4b35d7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kr46d" Oct 30 19:37:01.120698 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:37:01.120679 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de3476ef-83a0-4ca2-9b76-227ee4b35d7c-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-kr46d\" (UID: \"de3476ef-83a0-4ca2-9b76-227ee4b35d7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kr46d" Oct 30 19:37:01.121065 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:37:01.121046 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de3476ef-83a0-4ca2-9b76-227ee4b35d7c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-kr46d\" (UID: \"de3476ef-83a0-4ca2-9b76-227ee4b35d7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kr46d" Oct 30 19:37:01.122463 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:37:01.122446 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-q8znj\"" Oct 30 19:37:01.177868 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:37:01.177829 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de3476ef-83a0-4ca2-9b76-227ee4b35d7c-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-kr46d\" (UID: \"de3476ef-83a0-4ca2-9b76-227ee4b35d7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kr46d" Oct 30 19:37:01.178027 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:37:01.177908 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79l4d\" (UniqueName: \"kubernetes.io/projected/de3476ef-83a0-4ca2-9b76-227ee4b35d7c-kube-api-access-79l4d\") pod \"maas-default-gateway-openshift-default-694db55c87-kr46d\" (UID: \"de3476ef-83a0-4ca2-9b76-227ee4b35d7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kr46d" Oct 30 19:37:01.296771 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:37:01.296732 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kr46d" Oct 30 19:37:01.722915 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:37:01.722837 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kr46d"] Oct 30 19:37:01.729732 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:37:01.729704 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde3476ef_83a0_4ca2_9b76_227ee4b35d7c.slice/crio-f34b6db23573cfeec4dbf1665c4c30e3f6d7745525c38e35c5ef8b1290683066 WatchSource:0}: Error finding container f34b6db23573cfeec4dbf1665c4c30e3f6d7745525c38e35c5ef8b1290683066: Status 404 returned error can't find the container with id f34b6db23573cfeec4dbf1665c4c30e3f6d7745525c38e35c5ef8b1290683066 Oct 30 19:37:01.731635 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:37:01.731617 4292 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:37:02.249709 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:37:02.249664 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kr46d" event={"ID":"de3476ef-83a0-4ca2-9b76-227ee4b35d7c","Type":"ContainerStarted","Data":"f34b6db23573cfeec4dbf1665c4c30e3f6d7745525c38e35c5ef8b1290683066"} Oct 30 19:37:05.866120 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:37:05.866090 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:37:05.866422 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:37:05.866156 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:37:05.866422 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:37:05.866186 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:37:06.269948 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:37:06.269907 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kr46d" event={"ID":"de3476ef-83a0-4ca2-9b76-227ee4b35d7c","Type":"ContainerStarted","Data":"2fb856ce0abe7e7894144321855bd955ee318aee06031b6870882f8eab86b69c"} Oct 30 19:37:06.298837 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:37:06.298748 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kr46d" Oct 30 19:37:06.300643 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:37:06.300272 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-kr46d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Oct 30 19:37:06.300643 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:37:06.300338 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kr46d" podUID="de3476ef-83a0-4ca2-9b76-227ee4b35d7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Oct 30 19:37:06.467374 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:37:06.466785 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kr46d" podStartSLOduration=2.333296534 podStartE2EDuration="6.466765918s" totalImagesPullingTime="4.133469384s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:06.431474754 +0000 UTC m=+729.949790213" watchObservedRunningTime="2026-10-30 19:37:06.466765918 +0000 UTC m=+729.985081377" Oct 30 19:37:07.298029 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:37:07.297990 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-kr46d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Oct 30 19:37:07.298447 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:37:07.298045 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kr46d" podUID="de3476ef-83a0-4ca2-9b76-227ee4b35d7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Oct 30 19:37:08.297971 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:37:08.297933 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-kr46d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Oct 30 19:37:08.298160 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:37:08.297991 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kr46d" podUID="de3476ef-83a0-4ca2-9b76-227ee4b35d7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Oct 30 19:37:09.297326 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:37:09.297287 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-kr46d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Oct 30 19:37:09.297530 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:37:09.297343 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kr46d" podUID="de3476ef-83a0-4ca2-9b76-227ee4b35d7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Oct 30 19:37:10.297567 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:37:10.297529 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-kr46d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Oct 30 19:37:10.297972 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:37:10.297582 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kr46d" podUID="de3476ef-83a0-4ca2-9b76-227ee4b35d7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Oct 30 19:37:11.298168 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:37:11.298131 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-kr46d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Oct 30 19:37:11.298572 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:37:11.298212 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kr46d" podUID="de3476ef-83a0-4ca2-9b76-227ee4b35d7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Oct 30 19:37:12.297418 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:37:12.297384 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-kr46d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Oct 30 19:37:12.297626 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:37:12.297438 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kr46d" podUID="de3476ef-83a0-4ca2-9b76-227ee4b35d7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Oct 30 19:37:13.223936 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:37:13.223892 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:37:13.611275 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:37:13.611226 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kr46d" Oct 30 19:37:13.611899 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:37:13.611875 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kr46d" Oct 30 19:37:13.612640 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:37:13.612616 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kr46d" Oct 30 19:37:43.220573 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:37:43.220494 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:38:13.220918 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:38:13.220873 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:38:30.453390 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:30.453339 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f214434c-6bf6bf9c9-fdn7v"] Oct 30 19:38:30.456655 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:30.456633 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f214434c-6bf6bf9c9-fdn7v" Oct 30 19:38:30.510152 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:30.510117 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-f214434c\"" Oct 30 19:38:30.553774 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:30.553738 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-f214434c-dockercfg-vrgsc\"" Oct 30 19:38:30.556011 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:30.555982 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f214434c-76498c984b-zp2wr"] Oct 30 19:38:30.559408 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:30.559339 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f214434c-6bf6bf9c9-fdn7v"] Oct 30 19:38:30.559572 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:30.559480 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f214434c-76498c984b-zp2wr" Oct 30 19:38:30.574150 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:30.574121 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f214434c-76498c984b-zp2wr"] Oct 30 19:38:30.591072 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:30.591041 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf8c2615-ae01-4665-9ba2-b150b67a9f95-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-f214434c-6bf6bf9c9-fdn7v\" (UID: \"cf8c2615-ae01-4665-9ba2-b150b67a9f95\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f214434c-6bf6bf9c9-fdn7v" Oct 30 19:38:30.591260 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:30.591110 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqm5r\" (UniqueName: \"kubernetes.io/projected/cf8c2615-ae01-4665-9ba2-b150b67a9f95-kube-api-access-zqm5r\") pod \"payload-pre-processing-e2e-migrate-f214434c-6bf6bf9c9-fdn7v\" (UID: \"cf8c2615-ae01-4665-9ba2-b150b67a9f95\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f214434c-6bf6bf9c9-fdn7v" Oct 30 19:38:30.691935 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:30.691902 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f5dd40b-3f1c-411e-b62c-a0a9690743de-plugins-config-volume\") pod \"payload-processing-e2e-migrate-f214434c-76498c984b-zp2wr\" (UID: \"6f5dd40b-3f1c-411e-b62c-a0a9690743de\") " pod="openshift-ingress/payload-processing-e2e-migrate-f214434c-76498c984b-zp2wr" Oct 30 19:38:30.692126 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:30.691943 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqm5r\" (UniqueName: \"kubernetes.io/projected/cf8c2615-ae01-4665-9ba2-b150b67a9f95-kube-api-access-zqm5r\") pod \"payload-pre-processing-e2e-migrate-f214434c-6bf6bf9c9-fdn7v\" (UID: \"cf8c2615-ae01-4665-9ba2-b150b67a9f95\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f214434c-6bf6bf9c9-fdn7v" Oct 30 19:38:30.692126 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:30.692003 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxlzk\" (UniqueName: \"kubernetes.io/projected/6f5dd40b-3f1c-411e-b62c-a0a9690743de-kube-api-access-jxlzk\") pod \"payload-processing-e2e-migrate-f214434c-76498c984b-zp2wr\" (UID: \"6f5dd40b-3f1c-411e-b62c-a0a9690743de\") " pod="openshift-ingress/payload-processing-e2e-migrate-f214434c-76498c984b-zp2wr" Oct 30 19:38:30.692126 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:30.692052 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf8c2615-ae01-4665-9ba2-b150b67a9f95-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-f214434c-6bf6bf9c9-fdn7v\" (UID: \"cf8c2615-ae01-4665-9ba2-b150b67a9f95\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f214434c-6bf6bf9c9-fdn7v" Oct 30 19:38:30.692739 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:30.692718 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf8c2615-ae01-4665-9ba2-b150b67a9f95-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-f214434c-6bf6bf9c9-fdn7v\" (UID: \"cf8c2615-ae01-4665-9ba2-b150b67a9f95\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f214434c-6bf6bf9c9-fdn7v" Oct 30 19:38:30.751019 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:30.750947 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqm5r\" (UniqueName: \"kubernetes.io/projected/cf8c2615-ae01-4665-9ba2-b150b67a9f95-kube-api-access-zqm5r\") pod \"payload-pre-processing-e2e-migrate-f214434c-6bf6bf9c9-fdn7v\" (UID: \"cf8c2615-ae01-4665-9ba2-b150b67a9f95\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f214434c-6bf6bf9c9-fdn7v" Oct 30 19:38:30.769769 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:30.769738 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f214434c-6bf6bf9c9-fdn7v" Oct 30 19:38:30.793105 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:30.793066 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f5dd40b-3f1c-411e-b62c-a0a9690743de-plugins-config-volume\") pod \"payload-processing-e2e-migrate-f214434c-76498c984b-zp2wr\" (UID: \"6f5dd40b-3f1c-411e-b62c-a0a9690743de\") " pod="openshift-ingress/payload-processing-e2e-migrate-f214434c-76498c984b-zp2wr" Oct 30 19:38:30.793270 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:30.793125 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxlzk\" (UniqueName: \"kubernetes.io/projected/6f5dd40b-3f1c-411e-b62c-a0a9690743de-kube-api-access-jxlzk\") pod \"payload-processing-e2e-migrate-f214434c-76498c984b-zp2wr\" (UID: \"6f5dd40b-3f1c-411e-b62c-a0a9690743de\") " pod="openshift-ingress/payload-processing-e2e-migrate-f214434c-76498c984b-zp2wr" Oct 30 19:38:30.793966 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:30.793944 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f5dd40b-3f1c-411e-b62c-a0a9690743de-plugins-config-volume\") pod \"payload-processing-e2e-migrate-f214434c-76498c984b-zp2wr\" (UID: \"6f5dd40b-3f1c-411e-b62c-a0a9690743de\") " pod="openshift-ingress/payload-processing-e2e-migrate-f214434c-76498c984b-zp2wr" Oct 30 19:38:30.851888 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:30.851853 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxlzk\" (UniqueName: \"kubernetes.io/projected/6f5dd40b-3f1c-411e-b62c-a0a9690743de-kube-api-access-jxlzk\") pod \"payload-processing-e2e-migrate-f214434c-76498c984b-zp2wr\" (UID: \"6f5dd40b-3f1c-411e-b62c-a0a9690743de\") " pod="openshift-ingress/payload-processing-e2e-migrate-f214434c-76498c984b-zp2wr" Oct 30 19:38:30.873540 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:30.873508 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f214434c-76498c984b-zp2wr" Oct 30 19:38:31.149685 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:31.149651 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f214434c-6bf6bf9c9-fdn7v"] Oct 30 19:38:31.156632 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:38:31.156607 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf8c2615_ae01_4665_9ba2_b150b67a9f95.slice/crio-c8256b6f08786f3fab026084d89b12119e13755471a2d2f5d3ed173992665374 WatchSource:0}: Error finding container c8256b6f08786f3fab026084d89b12119e13755471a2d2f5d3ed173992665374: Status 404 returned error can't find the container with id c8256b6f08786f3fab026084d89b12119e13755471a2d2f5d3ed173992665374 Oct 30 19:38:31.268190 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:31.268152 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f214434c-76498c984b-zp2wr"] Oct 30 19:38:31.275042 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:38:31.275017 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f5dd40b_3f1c_411e_b62c_a0a9690743de.slice/crio-ddbea28fc12a511dacaf3c355566685058e060ec7b60b94f263bcff40a626d89 WatchSource:0}: Error finding container ddbea28fc12a511dacaf3c355566685058e060ec7b60b94f263bcff40a626d89: Status 404 returned error can't find the container with id ddbea28fc12a511dacaf3c355566685058e060ec7b60b94f263bcff40a626d89 Oct 30 19:38:31.628916 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:31.628877 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f214434c-6bf6bf9c9-fdn7v" event={"ID":"cf8c2615-ae01-4665-9ba2-b150b67a9f95","Type":"ContainerStarted","Data":"c8256b6f08786f3fab026084d89b12119e13755471a2d2f5d3ed173992665374"} Oct 30 19:38:31.630144 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:31.630121 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f214434c-76498c984b-zp2wr" event={"ID":"6f5dd40b-3f1c-411e-b62c-a0a9690743de","Type":"ContainerStarted","Data":"ddbea28fc12a511dacaf3c355566685058e060ec7b60b94f263bcff40a626d89"} Oct 30 19:38:34.647656 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:34.647613 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f214434c-76498c984b-zp2wr" event={"ID":"6f5dd40b-3f1c-411e-b62c-a0a9690743de","Type":"ContainerStarted","Data":"14b4f21a1ee52b1dc74463d1a786710a6c6c2d7162fcadad0334c137c56cc877"} Oct 30 19:38:34.648133 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:34.647721 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-f214434c-76498c984b-zp2wr" Oct 30 19:38:34.649132 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:34.649111 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f214434c-6bf6bf9c9-fdn7v" event={"ID":"cf8c2615-ae01-4665-9ba2-b150b67a9f95","Type":"ContainerStarted","Data":"10a5a2ba72d1ce58c2b4c27b6eef4355df6738df240786e467186cadb72f2dd0"} Oct 30 19:38:34.649210 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:34.649155 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f214434c-6bf6bf9c9-fdn7v" Oct 30 19:38:34.760141 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:34.760082 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-f214434c-76498c984b-zp2wr" podStartSLOduration=1.958329441 podStartE2EDuration="4.760069113s" totalImagesPullingTime="2.801739672s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:34.759249131 +0000 UTC m=+818.277564587" watchObservedRunningTime="2026-10-30 19:38:34.760069113 +0000 UTC m=+818.278384570" Oct 30 19:38:34.870821 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:34.870770 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f214434c-6bf6bf9c9-fdn7v" podStartSLOduration=2.009776064 podStartE2EDuration="4.870752635s" totalImagesPullingTime="2.860976571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:34.869674897 +0000 UTC m=+818.387990353" watchObservedRunningTime="2026-10-30 19:38:34.870752635 +0000 UTC m=+818.389068091" Oct 30 19:38:42.853052 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:42.853015 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-be15b010-784fb446ff-vvn26"] Oct 30 19:38:42.856566 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:42.856543 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-be15b010-784fb446ff-vvn26" Oct 30 19:38:42.913116 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:42.913073 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5988ef3c-80b4-47fa-bcb6-49f8b65204d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-be15b010-784fb446ff-vvn26\" (UID: \"5988ef3c-80b4-47fa-bcb6-49f8b65204d2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-be15b010-784fb446ff-vvn26" Oct 30 19:38:42.913116 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:42.913122 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7rsj\" (UniqueName: \"kubernetes.io/projected/5988ef3c-80b4-47fa-bcb6-49f8b65204d2-kube-api-access-t7rsj\") pod \"payload-pre-processing-e2e-derive-be15b010-784fb446ff-vvn26\" (UID: \"5988ef3c-80b4-47fa-bcb6-49f8b65204d2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-be15b010-784fb446ff-vvn26" Oct 30 19:38:42.952980 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:42.952935 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-be15b010-dockercfg-r8d9x\"" Oct 30 19:38:42.952980 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:42.952958 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-be15b010\"" Oct 30 19:38:42.954458 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:42.954431 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-be15b010-784fb446ff-vvn26"] Oct 30 19:38:43.001052 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:43.001020 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-be15b010-85c96c9ff6-bx7tr"] Oct 30 19:38:43.004336 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:43.004318 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-be15b010-85c96c9ff6-bx7tr" Oct 30 19:38:43.013685 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:43.013655 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5988ef3c-80b4-47fa-bcb6-49f8b65204d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-be15b010-784fb446ff-vvn26\" (UID: \"5988ef3c-80b4-47fa-bcb6-49f8b65204d2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-be15b010-784fb446ff-vvn26" Oct 30 19:38:43.013829 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:43.013693 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7rsj\" (UniqueName: \"kubernetes.io/projected/5988ef3c-80b4-47fa-bcb6-49f8b65204d2-kube-api-access-t7rsj\") pod \"payload-pre-processing-e2e-derive-be15b010-784fb446ff-vvn26\" (UID: \"5988ef3c-80b4-47fa-bcb6-49f8b65204d2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-be15b010-784fb446ff-vvn26" Oct 30 19:38:43.013829 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:43.013729 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87ce2c93-5462-48ff-ac79-813cfd1ad576-plugins-config-volume\") pod \"payload-processing-e2e-derive-be15b010-85c96c9ff6-bx7tr\" (UID: \"87ce2c93-5462-48ff-ac79-813cfd1ad576\") " pod="openshift-ingress/payload-processing-e2e-derive-be15b010-85c96c9ff6-bx7tr" Oct 30 19:38:43.013935 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:43.013865 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mt8z\" (UniqueName: \"kubernetes.io/projected/87ce2c93-5462-48ff-ac79-813cfd1ad576-kube-api-access-5mt8z\") pod \"payload-processing-e2e-derive-be15b010-85c96c9ff6-bx7tr\" (UID: \"87ce2c93-5462-48ff-ac79-813cfd1ad576\") " pod="openshift-ingress/payload-processing-e2e-derive-be15b010-85c96c9ff6-bx7tr" Oct 30 19:38:43.014433 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:43.014415 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5988ef3c-80b4-47fa-bcb6-49f8b65204d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-be15b010-784fb446ff-vvn26\" (UID: \"5988ef3c-80b4-47fa-bcb6-49f8b65204d2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-be15b010-784fb446ff-vvn26" Oct 30 19:38:43.051346 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:43.051302 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-be15b010-85c96c9ff6-bx7tr"] Oct 30 19:38:43.072498 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:43.072464 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7rsj\" (UniqueName: \"kubernetes.io/projected/5988ef3c-80b4-47fa-bcb6-49f8b65204d2-kube-api-access-t7rsj\") pod \"payload-pre-processing-e2e-derive-be15b010-784fb446ff-vvn26\" (UID: \"5988ef3c-80b4-47fa-bcb6-49f8b65204d2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-be15b010-784fb446ff-vvn26" Oct 30 19:38:43.114412 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:43.114323 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mt8z\" (UniqueName: \"kubernetes.io/projected/87ce2c93-5462-48ff-ac79-813cfd1ad576-kube-api-access-5mt8z\") pod \"payload-processing-e2e-derive-be15b010-85c96c9ff6-bx7tr\" (UID: \"87ce2c93-5462-48ff-ac79-813cfd1ad576\") " pod="openshift-ingress/payload-processing-e2e-derive-be15b010-85c96c9ff6-bx7tr" Oct 30 19:38:43.114561 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:43.114430 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87ce2c93-5462-48ff-ac79-813cfd1ad576-plugins-config-volume\") pod \"payload-processing-e2e-derive-be15b010-85c96c9ff6-bx7tr\" (UID: \"87ce2c93-5462-48ff-ac79-813cfd1ad576\") " pod="openshift-ingress/payload-processing-e2e-derive-be15b010-85c96c9ff6-bx7tr" Oct 30 19:38:43.115087 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:43.115068 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87ce2c93-5462-48ff-ac79-813cfd1ad576-plugins-config-volume\") pod \"payload-processing-e2e-derive-be15b010-85c96c9ff6-bx7tr\" (UID: \"87ce2c93-5462-48ff-ac79-813cfd1ad576\") " pod="openshift-ingress/payload-processing-e2e-derive-be15b010-85c96c9ff6-bx7tr" Oct 30 19:38:43.168474 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:43.168425 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-be15b010-784fb446ff-vvn26" Oct 30 19:38:43.173791 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:43.173768 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mt8z\" (UniqueName: \"kubernetes.io/projected/87ce2c93-5462-48ff-ac79-813cfd1ad576-kube-api-access-5mt8z\") pod \"payload-processing-e2e-derive-be15b010-85c96c9ff6-bx7tr\" (UID: \"87ce2c93-5462-48ff-ac79-813cfd1ad576\") " pod="openshift-ingress/payload-processing-e2e-derive-be15b010-85c96c9ff6-bx7tr" Oct 30 19:38:43.232712 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:38:43.232668 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:38:43.317052 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:43.317010 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-be15b010-85c96c9ff6-bx7tr" Oct 30 19:38:43.540090 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:43.540054 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-be15b010-784fb446ff-vvn26"] Oct 30 19:38:43.546831 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:38:43.546800 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5988ef3c_80b4_47fa_bcb6_49f8b65204d2.slice/crio-6c6292b12d52ba2086e5c89f181b7e899aad53f035e587129c44caf0891a20ff WatchSource:0}: Error finding container 6c6292b12d52ba2086e5c89f181b7e899aad53f035e587129c44caf0891a20ff: Status 404 returned error can't find the container with id 6c6292b12d52ba2086e5c89f181b7e899aad53f035e587129c44caf0891a20ff Oct 30 19:38:43.687406 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:43.687288 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-be15b010-784fb446ff-vvn26" event={"ID":"5988ef3c-80b4-47fa-bcb6-49f8b65204d2","Type":"ContainerStarted","Data":"49ea8cefd0b7b6ac2eeed9d311166999cf0f4bf8bd32d20bdcd2674504374575"} Oct 30 19:38:43.687406 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:43.687341 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-be15b010-784fb446ff-vvn26" event={"ID":"5988ef3c-80b4-47fa-bcb6-49f8b65204d2","Type":"ContainerStarted","Data":"6c6292b12d52ba2086e5c89f181b7e899aad53f035e587129c44caf0891a20ff"} Oct 30 19:38:43.687406 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:43.687371 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-be15b010-85c96c9ff6-bx7tr"] Oct 30 19:38:43.687406 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:43.687399 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-be15b010-784fb446ff-vvn26" Oct 30 19:38:43.694052 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:38:43.694020 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87ce2c93_5462_48ff_ac79_813cfd1ad576.slice/crio-f9cfd58f461c14f00739e4517db06bda6c57c7523bd6c9cba2fe129d95dcaed5 WatchSource:0}: Error finding container f9cfd58f461c14f00739e4517db06bda6c57c7523bd6c9cba2fe129d95dcaed5: Status 404 returned error can't find the container with id f9cfd58f461c14f00739e4517db06bda6c57c7523bd6c9cba2fe129d95dcaed5 Oct 30 19:38:43.799580 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:43.799520 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-be15b010-784fb446ff-vvn26" podStartSLOduration=1.799505597 podStartE2EDuration="1.799505597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:43.798764797 +0000 UTC m=+827.317080276" watchObservedRunningTime="2026-10-30 19:38:43.799505597 +0000 UTC m=+827.317821055" Oct 30 19:38:44.693616 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:44.693570 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-be15b010-85c96c9ff6-bx7tr" event={"ID":"87ce2c93-5462-48ff-ac79-813cfd1ad576","Type":"ContainerStarted","Data":"102eda7d4ff0382dfa5b4362c1d6bc884d356e316d51bc565e2a241e55caf123"} Oct 30 19:38:44.694032 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:44.693629 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-be15b010-85c96c9ff6-bx7tr" event={"ID":"87ce2c93-5462-48ff-ac79-813cfd1ad576","Type":"ContainerStarted","Data":"f9cfd58f461c14f00739e4517db06bda6c57c7523bd6c9cba2fe129d95dcaed5"} Oct 30 19:38:44.694032 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:44.693656 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-be15b010-85c96c9ff6-bx7tr" Oct 30 19:38:44.804681 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:44.804650 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f214434c-fc7ffd7d6-9wx8r"] Oct 30 19:38:44.808052 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:44.808027 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f214434c-fc7ffd7d6-9wx8r" Oct 30 19:38:44.830708 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:44.830675 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c6cm\" (UniqueName: \"kubernetes.io/projected/cea92f32-a45f-42ee-b0da-79fc9186ec58-kube-api-access-2c6cm\") pod \"maas-api-e2e-migrate-f214434c-fc7ffd7d6-9wx8r\" (UID: \"cea92f32-a45f-42ee-b0da-79fc9186ec58\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f214434c-fc7ffd7d6-9wx8r" Oct 30 19:38:44.830863 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:44.830732 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cea92f32-a45f-42ee-b0da-79fc9186ec58-metrics-certs\") pod \"maas-api-e2e-migrate-f214434c-fc7ffd7d6-9wx8r\" (UID: \"cea92f32-a45f-42ee-b0da-79fc9186ec58\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f214434c-fc7ffd7d6-9wx8r" Oct 30 19:38:44.830863 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:44.830803 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cea92f32-a45f-42ee-b0da-79fc9186ec58-openshift-service-ca\") pod \"maas-api-e2e-migrate-f214434c-fc7ffd7d6-9wx8r\" (UID: \"cea92f32-a45f-42ee-b0da-79fc9186ec58\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f214434c-fc7ffd7d6-9wx8r" Oct 30 19:38:44.830863 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:44.830855 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cea92f32-a45f-42ee-b0da-79fc9186ec58-maas-api-tls\") pod \"maas-api-e2e-migrate-f214434c-fc7ffd7d6-9wx8r\" (UID: \"cea92f32-a45f-42ee-b0da-79fc9186ec58\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f214434c-fc7ffd7d6-9wx8r" Oct 30 19:38:44.850745 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:44.850696 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-be15b010-85c96c9ff6-bx7tr" podStartSLOduration=2.850683803 podStartE2EDuration="2.850683803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:44.805881511 +0000 UTC m=+828.324196988" watchObservedRunningTime="2026-10-30 19:38:44.850683803 +0000 UTC m=+828.368999304" Oct 30 19:38:44.858121 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:44.858082 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:38:44.858293 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:44.858145 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:38:44.858293 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:44.858182 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-f214434c-serving-cert\"" Oct 30 19:38:44.858436 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:44.858334 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 19:38:44.858436 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:44.858423 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-skrr4\"" Oct 30 19:38:44.898484 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:44.898451 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f214434c-fc7ffd7d6-9wx8r"] Oct 30 19:38:44.931510 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:44.931479 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cea92f32-a45f-42ee-b0da-79fc9186ec58-maas-api-tls\") pod \"maas-api-e2e-migrate-f214434c-fc7ffd7d6-9wx8r\" (UID: \"cea92f32-a45f-42ee-b0da-79fc9186ec58\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f214434c-fc7ffd7d6-9wx8r" Oct 30 19:38:44.931720 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:44.931519 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c6cm\" (UniqueName: \"kubernetes.io/projected/cea92f32-a45f-42ee-b0da-79fc9186ec58-kube-api-access-2c6cm\") pod \"maas-api-e2e-migrate-f214434c-fc7ffd7d6-9wx8r\" (UID: \"cea92f32-a45f-42ee-b0da-79fc9186ec58\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f214434c-fc7ffd7d6-9wx8r" Oct 30 19:38:44.931720 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:44.931594 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cea92f32-a45f-42ee-b0da-79fc9186ec58-metrics-certs\") pod \"maas-api-e2e-migrate-f214434c-fc7ffd7d6-9wx8r\" (UID: \"cea92f32-a45f-42ee-b0da-79fc9186ec58\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f214434c-fc7ffd7d6-9wx8r" Oct 30 19:38:44.931720 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:44.931642 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cea92f32-a45f-42ee-b0da-79fc9186ec58-openshift-service-ca\") pod \"maas-api-e2e-migrate-f214434c-fc7ffd7d6-9wx8r\" (UID: \"cea92f32-a45f-42ee-b0da-79fc9186ec58\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f214434c-fc7ffd7d6-9wx8r" Oct 30 19:38:44.932007 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:38:44.931723 4292 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-f214434c-serving-cert: secret "maas-api-e2e-migrate-f214434c-serving-cert" not found Oct 30 19:38:44.932007 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:38:44.931788 4292 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cea92f32-a45f-42ee-b0da-79fc9186ec58-maas-api-tls podName:cea92f32-a45f-42ee-b0da-79fc9186ec58 nodeName:}" failed. No retries permitted until 2026-10-30 19:38:45.431771237 +0000 UTC m=+828.950086672 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cea92f32-a45f-42ee-b0da-79fc9186ec58-maas-api-tls") pod "maas-api-e2e-migrate-f214434c-fc7ffd7d6-9wx8r" (UID: "cea92f32-a45f-42ee-b0da-79fc9186ec58") : secret "maas-api-e2e-migrate-f214434c-serving-cert" not found Oct 30 19:38:44.932520 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:44.932495 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cea92f32-a45f-42ee-b0da-79fc9186ec58-openshift-service-ca\") pod \"maas-api-e2e-migrate-f214434c-fc7ffd7d6-9wx8r\" (UID: \"cea92f32-a45f-42ee-b0da-79fc9186ec58\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f214434c-fc7ffd7d6-9wx8r" Oct 30 19:38:44.934618 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:44.934596 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cea92f32-a45f-42ee-b0da-79fc9186ec58-metrics-certs\") pod \"maas-api-e2e-migrate-f214434c-fc7ffd7d6-9wx8r\" (UID: \"cea92f32-a45f-42ee-b0da-79fc9186ec58\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f214434c-fc7ffd7d6-9wx8r" Oct 30 19:38:44.992878 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:44.992812 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c6cm\" (UniqueName: \"kubernetes.io/projected/cea92f32-a45f-42ee-b0da-79fc9186ec58-kube-api-access-2c6cm\") pod \"maas-api-e2e-migrate-f214434c-fc7ffd7d6-9wx8r\" (UID: \"cea92f32-a45f-42ee-b0da-79fc9186ec58\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f214434c-fc7ffd7d6-9wx8r" Oct 30 19:38:45.436269 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:45.436231 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cea92f32-a45f-42ee-b0da-79fc9186ec58-maas-api-tls\") pod \"maas-api-e2e-migrate-f214434c-fc7ffd7d6-9wx8r\" (UID: \"cea92f32-a45f-42ee-b0da-79fc9186ec58\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f214434c-fc7ffd7d6-9wx8r" Oct 30 19:38:45.438711 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:45.438686 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cea92f32-a45f-42ee-b0da-79fc9186ec58-maas-api-tls\") pod \"maas-api-e2e-migrate-f214434c-fc7ffd7d6-9wx8r\" (UID: \"cea92f32-a45f-42ee-b0da-79fc9186ec58\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f214434c-fc7ffd7d6-9wx8r" Oct 30 19:38:45.656969 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:45.656939 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f214434c-6bf6bf9c9-fdn7v" Oct 30 19:38:45.657106 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:45.657013 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-f214434c-76498c984b-zp2wr" Oct 30 19:38:45.721613 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:45.721522 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f214434c-fc7ffd7d6-9wx8r" Oct 30 19:38:46.143717 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:46.143683 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f214434c-fc7ffd7d6-9wx8r"] Oct 30 19:38:46.151712 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:38:46.151678 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcea92f32_a45f_42ee_b0da_79fc9186ec58.slice/crio-b1475a72adb564af3aa3fe5a888f778c7c8f96f34952efda2d04e7ef251b73bd WatchSource:0}: Error finding container b1475a72adb564af3aa3fe5a888f778c7c8f96f34952efda2d04e7ef251b73bd: Status 404 returned error can't find the container with id b1475a72adb564af3aa3fe5a888f778c7c8f96f34952efda2d04e7ef251b73bd Oct 30 19:38:46.703286 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:46.703248 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f214434c-fc7ffd7d6-9wx8r" event={"ID":"cea92f32-a45f-42ee-b0da-79fc9186ec58","Type":"ContainerStarted","Data":"b1475a72adb564af3aa3fe5a888f778c7c8f96f34952efda2d04e7ef251b73bd"} Oct 30 19:38:47.709663 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:47.709630 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f214434c-fc7ffd7d6-9wx8r" event={"ID":"cea92f32-a45f-42ee-b0da-79fc9186ec58","Type":"ContainerStarted","Data":"bd6fb3586514167093e27ef7eb5de579256f2a444a03bc38be141a0f1a38513e"} Oct 30 19:38:47.710070 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:47.709754 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f214434c-fc7ffd7d6-9wx8r" Oct 30 19:38:47.828486 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:47.828422 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f214434c-fc7ffd7d6-9wx8r" podStartSLOduration=2.977003901 podStartE2EDuration="3.828407585s" totalImagesPullingTime="851.403684ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:47.827394998 +0000 UTC m=+831.345710457" watchObservedRunningTime="2026-10-30 19:38:47.828407585 +0000 UTC m=+831.346723041" Oct 30 19:38:53.718894 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:53.718862 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f214434c-fc7ffd7d6-9wx8r" Oct 30 19:38:54.696648 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:54.696619 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-be15b010-784fb446ff-vvn26" Oct 30 19:38:55.701369 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:38:55.701328 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-be15b010-85c96c9ff6-bx7tr" Oct 30 19:39:10.547101 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:10.547015 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3df7da3e-n9xsp"] Oct 30 19:39:10.555080 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:10.555058 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3df7da3e-n9xsp" Oct 30 19:39:10.606252 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:10.606222 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3df7da3e-n9xsp"] Oct 30 19:39:10.650558 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:10.650527 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:39:10.650894 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:10.650876 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\"" Oct 30 19:39:10.656290 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:10.656265 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9jm2\" (UniqueName: \"kubernetes.io/projected/fb6f38c0-4573-4376-a7c6-760a7bde3111-kube-api-access-t9jm2\") pod \"maas-api-revoke-keys-e2e-ait-3df7da3e-n9xsp\" (UID: \"fb6f38c0-4573-4376-a7c6-760a7bde3111\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3df7da3e-n9xsp" Oct 30 19:39:10.656386 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:10.656311 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb6f38c0-4573-4376-a7c6-760a7bde3111-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-3df7da3e-n9xsp\" (UID: \"fb6f38c0-4573-4376-a7c6-760a7bde3111\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3df7da3e-n9xsp" Oct 30 19:39:10.656386 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:10.656360 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb6f38c0-4573-4376-a7c6-760a7bde3111-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-3df7da3e-n9xsp\" (UID: \"fb6f38c0-4573-4376-a7c6-760a7bde3111\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3df7da3e-n9xsp" Oct 30 19:39:10.757033 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:10.756995 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb6f38c0-4573-4376-a7c6-760a7bde3111-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-3df7da3e-n9xsp\" (UID: \"fb6f38c0-4573-4376-a7c6-760a7bde3111\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3df7da3e-n9xsp" Oct 30 19:39:10.757243 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:10.757052 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb6f38c0-4573-4376-a7c6-760a7bde3111-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-3df7da3e-n9xsp\" (UID: \"fb6f38c0-4573-4376-a7c6-760a7bde3111\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3df7da3e-n9xsp" Oct 30 19:39:10.757243 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:10.757172 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9jm2\" (UniqueName: \"kubernetes.io/projected/fb6f38c0-4573-4376-a7c6-760a7bde3111-kube-api-access-t9jm2\") pod \"maas-api-revoke-keys-e2e-ait-3df7da3e-n9xsp\" (UID: \"fb6f38c0-4573-4376-a7c6-760a7bde3111\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3df7da3e-n9xsp" Oct 30 19:39:10.757808 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:10.757788 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb6f38c0-4573-4376-a7c6-760a7bde3111-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-3df7da3e-n9xsp\" (UID: \"fb6f38c0-4573-4376-a7c6-760a7bde3111\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3df7da3e-n9xsp" Oct 30 19:39:10.759531 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:10.759508 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb6f38c0-4573-4376-a7c6-760a7bde3111-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-3df7da3e-n9xsp\" (UID: \"fb6f38c0-4573-4376-a7c6-760a7bde3111\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3df7da3e-n9xsp" Oct 30 19:39:10.815589 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:10.815516 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9jm2\" (UniqueName: \"kubernetes.io/projected/fb6f38c0-4573-4376-a7c6-760a7bde3111-kube-api-access-t9jm2\") pod \"maas-api-revoke-keys-e2e-ait-3df7da3e-n9xsp\" (UID: \"fb6f38c0-4573-4376-a7c6-760a7bde3111\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3df7da3e-n9xsp" Oct 30 19:39:10.886045 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:10.886011 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3df7da3e-n9xsp" Oct 30 19:39:11.254589 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:11.254508 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3df7da3e-n9xsp"] Oct 30 19:39:11.261836 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:39:11.261796 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb6f38c0_4573_4376_a7c6_760a7bde3111.slice/crio-1439515d51f3b2a6dff264ce8efe17d6fd069c0d713305f176e5a13e26fc75bc WatchSource:0}: Error finding container 1439515d51f3b2a6dff264ce8efe17d6fd069c0d713305f176e5a13e26fc75bc: Status 404 returned error can't find the container with id 1439515d51f3b2a6dff264ce8efe17d6fd069c0d713305f176e5a13e26fc75bc Oct 30 19:39:11.813532 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:11.813497 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3df7da3e-n9xsp" event={"ID":"fb6f38c0-4573-4376-a7c6-760a7bde3111","Type":"ContainerStarted","Data":"1439515d51f3b2a6dff264ce8efe17d6fd069c0d713305f176e5a13e26fc75bc"} Oct 30 19:39:13.220841 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:39:13.220804 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:39:15.833892 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:15.833857 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3df7da3e-n9xsp" event={"ID":"fb6f38c0-4573-4376-a7c6-760a7bde3111","Type":"ContainerStarted","Data":"ecb1b79e969368190adebc673e9e94667e6413b22fda42183bb3ea5fe00b05d2"} Oct 30 19:39:15.946224 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:15.946169 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3df7da3e-n9xsp" podStartSLOduration=2.282306695 podStartE2EDuration="5.946151798s" totalImagesPullingTime="3.663845103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:15.944393444 +0000 UTC m=+859.462708898" watchObservedRunningTime="2026-10-30 19:39:15.946151798 +0000 UTC m=+859.464467256" Oct 30 19:39:16.839393 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:16.839361 4292 generic.go:356] "Generic (PLEG): container finished" podID="fb6f38c0-4573-4376-a7c6-760a7bde3111" containerID="ecb1b79e969368190adebc673e9e94667e6413b22fda42183bb3ea5fe00b05d2" exitCode=0 Oct 30 19:39:16.839393 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:16.839378 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3df7da3e-n9xsp" event={"ID":"fb6f38c0-4573-4376-a7c6-760a7bde3111","Type":"ContainerDied","Data":"ecb1b79e969368190adebc673e9e94667e6413b22fda42183bb3ea5fe00b05d2"} Oct 30 19:39:17.984207 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:17.984174 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3df7da3e-n9xsp" Oct 30 19:39:18.124390 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:18.124276 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb6f38c0-4573-4376-a7c6-760a7bde3111-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb6f38c0-4573-4376-a7c6-760a7bde3111-maas-api-xks-ca\") pod \"fb6f38c0-4573-4376-a7c6-760a7bde3111\" (UID: \"fb6f38c0-4573-4376-a7c6-760a7bde3111\") " Oct 30 19:39:18.124390 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:18.124344 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb6f38c0-4573-4376-a7c6-760a7bde3111-kube-api-access-t9jm2\" (UniqueName: \"kubernetes.io/projected/fb6f38c0-4573-4376-a7c6-760a7bde3111-kube-api-access-t9jm2\") pod \"fb6f38c0-4573-4376-a7c6-760a7bde3111\" (UID: \"fb6f38c0-4573-4376-a7c6-760a7bde3111\") " Oct 30 19:39:18.124598 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:18.124486 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb6f38c0-4573-4376-a7c6-760a7bde3111-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb6f38c0-4573-4376-a7c6-760a7bde3111-maas-api-openshift-ca\") pod \"fb6f38c0-4573-4376-a7c6-760a7bde3111\" (UID: \"fb6f38c0-4573-4376-a7c6-760a7bde3111\") " Oct 30 19:39:18.124876 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:18.124849 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb6f38c0-4573-4376-a7c6-760a7bde3111-maas-api-openshift-ca" pod "fb6f38c0-4573-4376-a7c6-760a7bde3111" (UID: "fb6f38c0-4573-4376-a7c6-760a7bde3111"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:39:18.126523 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:18.126503 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb6f38c0-4573-4376-a7c6-760a7bde3111-maas-api-xks-ca" pod "fb6f38c0-4573-4376-a7c6-760a7bde3111" (UID: "fb6f38c0-4573-4376-a7c6-760a7bde3111"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:39:18.126600 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:18.126542 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb6f38c0-4573-4376-a7c6-760a7bde3111-kube-api-access-t9jm2" pod "fb6f38c0-4573-4376-a7c6-760a7bde3111" (UID: "fb6f38c0-4573-4376-a7c6-760a7bde3111"). InnerVolumeSpecName "kube-api-access-t9jm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:39:18.225313 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:18.225276 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb6f38c0-4573-4376-a7c6-760a7bde3111-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:39:18.225313 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:18.225309 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb6f38c0-4573-4376-a7c6-760a7bde3111-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:39:18.225313 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:18.225319 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9jm2\" (UniqueName: \"kubernetes.io/projected/fb6f38c0-4573-4376-a7c6-760a7bde3111-kube-api-access-t9jm2\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:39:18.849648 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:18.849565 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3df7da3e-n9xsp" event={"ID":"fb6f38c0-4573-4376-a7c6-760a7bde3111","Type":"ContainerDied","Data":"1439515d51f3b2a6dff264ce8efe17d6fd069c0d713305f176e5a13e26fc75bc"} Oct 30 19:39:18.849648 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:18.849600 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3df7da3e-n9xsp" Oct 30 19:39:18.849648 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:18.849615 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1439515d51f3b2a6dff264ce8efe17d6fd069c0d713305f176e5a13e26fc75bc" Oct 30 19:39:32.962901 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:32.962852 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f214434c-76498c984b-zp2wr"] Oct 30 19:39:33.126226 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:33.126185 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f214434c-6bf6bf9c9-fdn7v"] Oct 30 19:39:37.963606 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:37.963556 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-f214434c-76498c984b-zp2wr" podUID="6f5dd40b-3f1c-411e-b62c-a0a9690743de" containerName="payload-processing" containerID="cri-o://14b4f21a1ee52b1dc74463d1a786710a6c6c2d7162fcadad0334c137c56cc877" gracePeriod=25 Oct 30 19:39:38.126523 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:38.126468 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f214434c-6bf6bf9c9-fdn7v" podUID="cf8c2615-ae01-4665-9ba2-b150b67a9f95" containerName="payload-pre-processing" containerID="cri-o://10a5a2ba72d1ce58c2b4c27b6eef4355df6738df240786e467186cadb72f2dd0" gracePeriod=25 Oct 30 19:39:38.249806 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:38.249782 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f214434c-76498c984b-zp2wr" Oct 30 19:39:38.359930 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:38.359901 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f214434c-6bf6bf9c9-fdn7v" Oct 30 19:39:38.407050 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:38.407020 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f5dd40b-3f1c-411e-b62c-a0a9690743de-kube-api-access-jxlzk\" (UniqueName: \"kubernetes.io/projected/6f5dd40b-3f1c-411e-b62c-a0a9690743de-kube-api-access-jxlzk\") pod \"6f5dd40b-3f1c-411e-b62c-a0a9690743de\" (UID: \"6f5dd40b-3f1c-411e-b62c-a0a9690743de\") " Oct 30 19:39:38.407253 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:38.407071 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f5dd40b-3f1c-411e-b62c-a0a9690743de-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f5dd40b-3f1c-411e-b62c-a0a9690743de-plugins-config-volume\") pod \"6f5dd40b-3f1c-411e-b62c-a0a9690743de\" (UID: \"6f5dd40b-3f1c-411e-b62c-a0a9690743de\") " Oct 30 19:39:38.407502 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:38.407477 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f5dd40b-3f1c-411e-b62c-a0a9690743de-plugins-config-volume" pod "6f5dd40b-3f1c-411e-b62c-a0a9690743de" (UID: "6f5dd40b-3f1c-411e-b62c-a0a9690743de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:39:38.409107 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:38.409085 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f5dd40b-3f1c-411e-b62c-a0a9690743de-kube-api-access-jxlzk" pod "6f5dd40b-3f1c-411e-b62c-a0a9690743de" (UID: "6f5dd40b-3f1c-411e-b62c-a0a9690743de"). InnerVolumeSpecName "kube-api-access-jxlzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:39:38.508403 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:38.508293 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf8c2615-ae01-4665-9ba2-b150b67a9f95-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf8c2615-ae01-4665-9ba2-b150b67a9f95-plugins-config-volume\") pod \"cf8c2615-ae01-4665-9ba2-b150b67a9f95\" (UID: \"cf8c2615-ae01-4665-9ba2-b150b67a9f95\") " Oct 30 19:39:38.508403 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:38.508395 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf8c2615-ae01-4665-9ba2-b150b67a9f95-kube-api-access-zqm5r\" (UniqueName: \"kubernetes.io/projected/cf8c2615-ae01-4665-9ba2-b150b67a9f95-kube-api-access-zqm5r\") pod \"cf8c2615-ae01-4665-9ba2-b150b67a9f95\" (UID: \"cf8c2615-ae01-4665-9ba2-b150b67a9f95\") " Oct 30 19:39:38.508588 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:38.508559 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxlzk\" (UniqueName: \"kubernetes.io/projected/6f5dd40b-3f1c-411e-b62c-a0a9690743de-kube-api-access-jxlzk\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:39:38.508588 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:38.508571 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f5dd40b-3f1c-411e-b62c-a0a9690743de-plugins-config-volume\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:39:38.508805 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:38.508777 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf8c2615-ae01-4665-9ba2-b150b67a9f95-plugins-config-volume" pod "cf8c2615-ae01-4665-9ba2-b150b67a9f95" (UID: "cf8c2615-ae01-4665-9ba2-b150b67a9f95"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:39:38.510544 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:38.510521 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf8c2615-ae01-4665-9ba2-b150b67a9f95-kube-api-access-zqm5r" pod "cf8c2615-ae01-4665-9ba2-b150b67a9f95" (UID: "cf8c2615-ae01-4665-9ba2-b150b67a9f95"). InnerVolumeSpecName "kube-api-access-zqm5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:39:38.609591 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:38.609559 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf8c2615-ae01-4665-9ba2-b150b67a9f95-plugins-config-volume\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:39:38.609591 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:38.609587 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqm5r\" (UniqueName: \"kubernetes.io/projected/cf8c2615-ae01-4665-9ba2-b150b67a9f95-kube-api-access-zqm5r\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:39:38.937246 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:38.937201 4292 generic.go:356] "Generic (PLEG): container finished" podID="6f5dd40b-3f1c-411e-b62c-a0a9690743de" containerID="14b4f21a1ee52b1dc74463d1a786710a6c6c2d7162fcadad0334c137c56cc877" exitCode=0 Oct 30 19:39:38.937474 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:38.937262 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f214434c-76498c984b-zp2wr" Oct 30 19:39:38.937474 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:38.937290 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f214434c-76498c984b-zp2wr" event={"ID":"6f5dd40b-3f1c-411e-b62c-a0a9690743de","Type":"ContainerDied","Data":"14b4f21a1ee52b1dc74463d1a786710a6c6c2d7162fcadad0334c137c56cc877"} Oct 30 19:39:38.937474 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:38.937369 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f214434c-76498c984b-zp2wr" event={"ID":"6f5dd40b-3f1c-411e-b62c-a0a9690743de","Type":"ContainerDied","Data":"ddbea28fc12a511dacaf3c355566685058e060ec7b60b94f263bcff40a626d89"} Oct 30 19:39:38.937474 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:38.937400 4292 scope.go:122] "RemoveContainer" containerID="14b4f21a1ee52b1dc74463d1a786710a6c6c2d7162fcadad0334c137c56cc877" Oct 30 19:39:38.938608 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:38.938585 4292 generic.go:356] "Generic (PLEG): container finished" podID="cf8c2615-ae01-4665-9ba2-b150b67a9f95" containerID="10a5a2ba72d1ce58c2b4c27b6eef4355df6738df240786e467186cadb72f2dd0" exitCode=0 Oct 30 19:39:38.938684 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:38.938628 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f214434c-6bf6bf9c9-fdn7v" Oct 30 19:39:38.938684 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:38.938632 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f214434c-6bf6bf9c9-fdn7v" event={"ID":"cf8c2615-ae01-4665-9ba2-b150b67a9f95","Type":"ContainerDied","Data":"10a5a2ba72d1ce58c2b4c27b6eef4355df6738df240786e467186cadb72f2dd0"} Oct 30 19:39:38.938684 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:38.938680 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f214434c-6bf6bf9c9-fdn7v" event={"ID":"cf8c2615-ae01-4665-9ba2-b150b67a9f95","Type":"ContainerDied","Data":"c8256b6f08786f3fab026084d89b12119e13755471a2d2f5d3ed173992665374"} Oct 30 19:39:38.945919 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:38.945883 4292 scope.go:122] "RemoveContainer" containerID="14b4f21a1ee52b1dc74463d1a786710a6c6c2d7162fcadad0334c137c56cc877" Oct 30 19:39:38.946229 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:39:38.946206 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14b4f21a1ee52b1dc74463d1a786710a6c6c2d7162fcadad0334c137c56cc877\": container with ID starting with 14b4f21a1ee52b1dc74463d1a786710a6c6c2d7162fcadad0334c137c56cc877 not found: ID does not exist" containerID="14b4f21a1ee52b1dc74463d1a786710a6c6c2d7162fcadad0334c137c56cc877" Oct 30 19:39:38.946293 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:38.946239 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14b4f21a1ee52b1dc74463d1a786710a6c6c2d7162fcadad0334c137c56cc877"} err="failed to get container status \"14b4f21a1ee52b1dc74463d1a786710a6c6c2d7162fcadad0334c137c56cc877\": rpc error: code = NotFound desc = could not find container \"14b4f21a1ee52b1dc74463d1a786710a6c6c2d7162fcadad0334c137c56cc877\": container with ID starting with 14b4f21a1ee52b1dc74463d1a786710a6c6c2d7162fcadad0334c137c56cc877 not found: ID does not exist" Oct 30 19:39:38.946293 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:38.946258 4292 scope.go:122] "RemoveContainer" containerID="10a5a2ba72d1ce58c2b4c27b6eef4355df6738df240786e467186cadb72f2dd0" Oct 30 19:39:38.953911 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:38.953884 4292 scope.go:122] "RemoveContainer" containerID="10a5a2ba72d1ce58c2b4c27b6eef4355df6738df240786e467186cadb72f2dd0" Oct 30 19:39:38.954165 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:39:38.954148 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10a5a2ba72d1ce58c2b4c27b6eef4355df6738df240786e467186cadb72f2dd0\": container with ID starting with 10a5a2ba72d1ce58c2b4c27b6eef4355df6738df240786e467186cadb72f2dd0 not found: ID does not exist" containerID="10a5a2ba72d1ce58c2b4c27b6eef4355df6738df240786e467186cadb72f2dd0" Oct 30 19:39:38.954225 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:38.954170 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10a5a2ba72d1ce58c2b4c27b6eef4355df6738df240786e467186cadb72f2dd0"} err="failed to get container status \"10a5a2ba72d1ce58c2b4c27b6eef4355df6738df240786e467186cadb72f2dd0\": rpc error: code = NotFound desc = could not find container \"10a5a2ba72d1ce58c2b4c27b6eef4355df6738df240786e467186cadb72f2dd0\": container with ID starting with 10a5a2ba72d1ce58c2b4c27b6eef4355df6738df240786e467186cadb72f2dd0 not found: ID does not exist" Oct 30 19:39:39.100768 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:39.100721 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f214434c-6bf6bf9c9-fdn7v"] Oct 30 19:39:39.147463 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:39.147425 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f214434c-6bf6bf9c9-fdn7v"] Oct 30 19:39:39.254984 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:39.254874 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f214434c-76498c984b-zp2wr"] Oct 30 19:39:39.256957 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:39.256926 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f214434c-76498c984b-zp2wr"] Oct 30 19:39:40.581023 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:40.580990 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f5dd40b-3f1c-411e-b62c-a0a9690743de" path="/var/lib/kubelet/pods/6f5dd40b-3f1c-411e-b62c-a0a9690743de/volumes" Oct 30 19:39:40.581511 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:40.581496 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf8c2615-ae01-4665-9ba2-b150b67a9f95" path="/var/lib/kubelet/pods/cf8c2615-ae01-4665-9ba2-b150b67a9f95/volumes" Oct 30 19:39:40.626519 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:40.623061 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-be15b010-85c96c9ff6-bx7tr"] Oct 30 19:39:40.736934 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:40.736896 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-be15b010-784fb446ff-vvn26"] Oct 30 19:39:43.080140 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:43.080104 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3df7da3e-n9xsp"] Oct 30 19:39:43.127010 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:43.126968 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3df7da3e-n9xsp"] Oct 30 19:39:43.220885 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:39:43.220771 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:39:44.582170 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:44.582138 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb6f38c0-4573-4376-a7c6-760a7bde3111" path="/var/lib/kubelet/pods/fb6f38c0-4573-4376-a7c6-760a7bde3111/volumes" Oct 30 19:39:45.624140 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:45.624096 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-be15b010-85c96c9ff6-bx7tr" podUID="87ce2c93-5462-48ff-ac79-813cfd1ad576" containerName="payload-processing" containerID="cri-o://102eda7d4ff0382dfa5b4362c1d6bc884d356e316d51bc565e2a241e55caf123" gracePeriod=25 Oct 30 19:39:45.699617 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:39:45.699587 4292 logging.go:55] [core] [Channel #89 SubChannel #90]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.34:50052", ServerName: "10.133.0.34:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.34:50052: connect: connection refused" Oct 30 19:39:45.737809 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:45.737747 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-be15b010-784fb446ff-vvn26" podUID="5988ef3c-80b4-47fa-bcb6-49f8b65204d2" containerName="payload-pre-processing" containerID="cri-o://49ea8cefd0b7b6ac2eeed9d311166999cf0f4bf8bd32d20bdcd2674504374575" gracePeriod=25 Oct 30 19:39:45.879580 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:45.879519 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-be15b010-85c96c9ff6-bx7tr" Oct 30 19:39:45.967785 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:45.967752 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87ce2c93-5462-48ff-ac79-813cfd1ad576-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87ce2c93-5462-48ff-ac79-813cfd1ad576-plugins-config-volume\") pod \"87ce2c93-5462-48ff-ac79-813cfd1ad576\" (UID: \"87ce2c93-5462-48ff-ac79-813cfd1ad576\") " Oct 30 19:39:45.967965 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:45.967871 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87ce2c93-5462-48ff-ac79-813cfd1ad576-kube-api-access-5mt8z\" (UniqueName: \"kubernetes.io/projected/87ce2c93-5462-48ff-ac79-813cfd1ad576-kube-api-access-5mt8z\") pod \"87ce2c93-5462-48ff-ac79-813cfd1ad576\" (UID: \"87ce2c93-5462-48ff-ac79-813cfd1ad576\") " Oct 30 19:39:45.968251 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:45.968226 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87ce2c93-5462-48ff-ac79-813cfd1ad576-plugins-config-volume" pod "87ce2c93-5462-48ff-ac79-813cfd1ad576" (UID: "87ce2c93-5462-48ff-ac79-813cfd1ad576"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:39:45.970162 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:45.970133 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87ce2c93-5462-48ff-ac79-813cfd1ad576-kube-api-access-5mt8z" pod "87ce2c93-5462-48ff-ac79-813cfd1ad576" (UID: "87ce2c93-5462-48ff-ac79-813cfd1ad576"). InnerVolumeSpecName "kube-api-access-5mt8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:39:45.972791 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:45.972768 4292 generic.go:356] "Generic (PLEG): container finished" podID="5988ef3c-80b4-47fa-bcb6-49f8b65204d2" containerID="49ea8cefd0b7b6ac2eeed9d311166999cf0f4bf8bd32d20bdcd2674504374575" exitCode=0 Oct 30 19:39:45.972899 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:45.972832 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-be15b010-784fb446ff-vvn26" event={"ID":"5988ef3c-80b4-47fa-bcb6-49f8b65204d2","Type":"ContainerDied","Data":"49ea8cefd0b7b6ac2eeed9d311166999cf0f4bf8bd32d20bdcd2674504374575"} Oct 30 19:39:45.974243 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:45.974223 4292 generic.go:356] "Generic (PLEG): container finished" podID="87ce2c93-5462-48ff-ac79-813cfd1ad576" containerID="102eda7d4ff0382dfa5b4362c1d6bc884d356e316d51bc565e2a241e55caf123" exitCode=0 Oct 30 19:39:45.974295 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:45.974270 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-be15b010-85c96c9ff6-bx7tr" Oct 30 19:39:45.974378 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:45.974272 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-be15b010-85c96c9ff6-bx7tr" event={"ID":"87ce2c93-5462-48ff-ac79-813cfd1ad576","Type":"ContainerDied","Data":"102eda7d4ff0382dfa5b4362c1d6bc884d356e316d51bc565e2a241e55caf123"} Oct 30 19:39:45.974450 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:45.974412 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-be15b010-85c96c9ff6-bx7tr" event={"ID":"87ce2c93-5462-48ff-ac79-813cfd1ad576","Type":"ContainerDied","Data":"f9cfd58f461c14f00739e4517db06bda6c57c7523bd6c9cba2fe129d95dcaed5"} Oct 30 19:39:45.974504 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:45.974443 4292 scope.go:122] "RemoveContainer" containerID="102eda7d4ff0382dfa5b4362c1d6bc884d356e316d51bc565e2a241e55caf123" Oct 30 19:39:45.985399 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:45.985381 4292 scope.go:122] "RemoveContainer" containerID="102eda7d4ff0382dfa5b4362c1d6bc884d356e316d51bc565e2a241e55caf123" Oct 30 19:39:45.985656 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:39:45.985638 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"102eda7d4ff0382dfa5b4362c1d6bc884d356e316d51bc565e2a241e55caf123\": container with ID starting with 102eda7d4ff0382dfa5b4362c1d6bc884d356e316d51bc565e2a241e55caf123 not found: ID does not exist" containerID="102eda7d4ff0382dfa5b4362c1d6bc884d356e316d51bc565e2a241e55caf123" Oct 30 19:39:45.985716 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:45.985663 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"102eda7d4ff0382dfa5b4362c1d6bc884d356e316d51bc565e2a241e55caf123"} err="failed to get container status \"102eda7d4ff0382dfa5b4362c1d6bc884d356e316d51bc565e2a241e55caf123\": rpc error: code = NotFound desc = could not find container \"102eda7d4ff0382dfa5b4362c1d6bc884d356e316d51bc565e2a241e55caf123\": container with ID starting with 102eda7d4ff0382dfa5b4362c1d6bc884d356e316d51bc565e2a241e55caf123 not found: ID does not exist" Oct 30 19:39:45.985759 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:45.985726 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-be15b010-784fb446ff-vvn26" Oct 30 19:39:46.069142 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:46.069101 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5988ef3c-80b4-47fa-bcb6-49f8b65204d2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5988ef3c-80b4-47fa-bcb6-49f8b65204d2-plugins-config-volume\") pod \"5988ef3c-80b4-47fa-bcb6-49f8b65204d2\" (UID: \"5988ef3c-80b4-47fa-bcb6-49f8b65204d2\") " Oct 30 19:39:46.069142 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:46.069157 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5988ef3c-80b4-47fa-bcb6-49f8b65204d2-kube-api-access-t7rsj\" (UniqueName: \"kubernetes.io/projected/5988ef3c-80b4-47fa-bcb6-49f8b65204d2-kube-api-access-t7rsj\") pod \"5988ef3c-80b4-47fa-bcb6-49f8b65204d2\" (UID: \"5988ef3c-80b4-47fa-bcb6-49f8b65204d2\") " Oct 30 19:39:46.069421 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:46.069370 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87ce2c93-5462-48ff-ac79-813cfd1ad576-plugins-config-volume\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:39:46.069421 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:46.069383 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mt8z\" (UniqueName: \"kubernetes.io/projected/87ce2c93-5462-48ff-ac79-813cfd1ad576-kube-api-access-5mt8z\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:39:46.069540 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:46.069516 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5988ef3c-80b4-47fa-bcb6-49f8b65204d2-plugins-config-volume" pod "5988ef3c-80b4-47fa-bcb6-49f8b65204d2" (UID: "5988ef3c-80b4-47fa-bcb6-49f8b65204d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:39:46.071240 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:46.071221 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5988ef3c-80b4-47fa-bcb6-49f8b65204d2-kube-api-access-t7rsj" pod "5988ef3c-80b4-47fa-bcb6-49f8b65204d2" (UID: "5988ef3c-80b4-47fa-bcb6-49f8b65204d2"). InnerVolumeSpecName "kube-api-access-t7rsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:39:46.170184 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:46.170095 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5988ef3c-80b4-47fa-bcb6-49f8b65204d2-plugins-config-volume\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:39:46.170184 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:46.170129 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7rsj\" (UniqueName: \"kubernetes.io/projected/5988ef3c-80b4-47fa-bcb6-49f8b65204d2-kube-api-access-t7rsj\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:39:46.205998 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:46.205959 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-be15b010-85c96c9ff6-bx7tr"] Oct 30 19:39:46.213465 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:46.213434 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-be15b010-85c96c9ff6-bx7tr"] Oct 30 19:39:46.585908 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:46.585876 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87ce2c93-5462-48ff-ac79-813cfd1ad576" path="/var/lib/kubelet/pods/87ce2c93-5462-48ff-ac79-813cfd1ad576/volumes" Oct 30 19:39:46.699525 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:46.699479 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-derive-be15b010-85c96c9ff6-bx7tr" podUID="87ce2c93-5462-48ff-ac79-813cfd1ad576" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.34:50052\" within 1s: context deadline exceeded" Oct 30 19:39:46.979987 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:46.979899 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-be15b010-784fb446ff-vvn26" Oct 30 19:39:46.979987 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:46.979918 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-be15b010-784fb446ff-vvn26" event={"ID":"5988ef3c-80b4-47fa-bcb6-49f8b65204d2","Type":"ContainerDied","Data":"6c6292b12d52ba2086e5c89f181b7e899aad53f035e587129c44caf0891a20ff"} Oct 30 19:39:46.980225 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:46.979995 4292 scope.go:122] "RemoveContainer" containerID="49ea8cefd0b7b6ac2eeed9d311166999cf0f4bf8bd32d20bdcd2674504374575" Oct 30 19:39:47.135624 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:47.135586 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-be15b010-784fb446ff-vvn26"] Oct 30 19:39:47.182304 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:47.182265 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-be15b010-784fb446ff-vvn26"] Oct 30 19:39:48.580809 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:48.580776 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5988ef3c-80b4-47fa-bcb6-49f8b65204d2" path="/var/lib/kubelet/pods/5988ef3c-80b4-47fa-bcb6-49f8b65204d2/volumes" Oct 30 19:39:52.451329 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:52.451286 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ec609-754d6f868c-l66gq"] Oct 30 19:39:52.451853 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:52.451760 4292 cpu_manager.go:414] "containerMap: removing container" podUID="5988ef3c-80b4-47fa-bcb6-49f8b65204d2" containerName="payload-pre-processing" Oct 30 19:39:52.451853 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:52.451779 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5988ef3c-80b4-47fa-bcb6-49f8b65204d2" containerName="payload-pre-processing" Oct 30 19:39:52.451853 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:52.451805 4292 cpu_manager.go:414] "containerMap: removing container" podUID="6f5dd40b-3f1c-411e-b62c-a0a9690743de" containerName="payload-processing" Oct 30 19:39:52.451853 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:52.451814 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f5dd40b-3f1c-411e-b62c-a0a9690743de" containerName="payload-processing" Oct 30 19:39:52.451853 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:52.451824 4292 cpu_manager.go:414] "containerMap: removing container" podUID="87ce2c93-5462-48ff-ac79-813cfd1ad576" containerName="payload-processing" Oct 30 19:39:52.451853 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:52.451831 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87ce2c93-5462-48ff-ac79-813cfd1ad576" containerName="payload-processing" Oct 30 19:39:52.451853 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:52.451852 4292 cpu_manager.go:414] "containerMap: removing container" podUID="cf8c2615-ae01-4665-9ba2-b150b67a9f95" containerName="payload-pre-processing" Oct 30 19:39:52.452201 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:52.451860 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf8c2615-ae01-4665-9ba2-b150b67a9f95" containerName="payload-pre-processing" Oct 30 19:39:52.452201 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:52.451869 4292 cpu_manager.go:414] "containerMap: removing container" podUID="fb6f38c0-4573-4376-a7c6-760a7bde3111" containerName="revoke-keys" Oct 30 19:39:52.452201 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:52.451876 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb6f38c0-4573-4376-a7c6-760a7bde3111" containerName="revoke-keys" Oct 30 19:39:52.452201 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:52.451969 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="5988ef3c-80b4-47fa-bcb6-49f8b65204d2" containerName="payload-pre-processing" Oct 30 19:39:52.452201 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:52.451982 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f5dd40b-3f1c-411e-b62c-a0a9690743de" containerName="payload-processing" Oct 30 19:39:52.452201 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:52.451992 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="87ce2c93-5462-48ff-ac79-813cfd1ad576" containerName="payload-processing" Oct 30 19:39:52.452201 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:52.452002 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf8c2615-ae01-4665-9ba2-b150b67a9f95" containerName="payload-pre-processing" Oct 30 19:39:52.452201 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:52.452012 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb6f38c0-4573-4376-a7c6-760a7bde3111" containerName="revoke-keys" Oct 30 19:39:52.457996 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:52.457972 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ec609-754d6f868c-l66gq" Oct 30 19:39:52.524951 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:52.524918 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85408ab0-d248-46df-a359-6a213c10fd58-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-4ec609-754d6f868c-l66gq\" (UID: \"85408ab0-d248-46df-a359-6a213c10fd58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ec609-754d6f868c-l66gq" Oct 30 19:39:52.525120 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:52.524969 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85408ab0-d248-46df-a359-6a213c10fd58-metrics-certs\") pod \"maas-api-e2e-worker-w4-4ec609-754d6f868c-l66gq\" (UID: \"85408ab0-d248-46df-a359-6a213c10fd58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ec609-754d6f868c-l66gq" Oct 30 19:39:52.525120 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:52.525097 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85408ab0-d248-46df-a359-6a213c10fd58-maas-api-tls\") pod \"maas-api-e2e-worker-w4-4ec609-754d6f868c-l66gq\" (UID: \"85408ab0-d248-46df-a359-6a213c10fd58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ec609-754d6f868c-l66gq" Oct 30 19:39:52.525206 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:52.525127 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4r5l\" (UniqueName: \"kubernetes.io/projected/85408ab0-d248-46df-a359-6a213c10fd58-kube-api-access-q4r5l\") pod \"maas-api-e2e-worker-w4-4ec609-754d6f868c-l66gq\" (UID: \"85408ab0-d248-46df-a359-6a213c10fd58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ec609-754d6f868c-l66gq" Oct 30 19:39:52.556437 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:52.556405 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-4ec609-serving-cert\"" Oct 30 19:39:52.557943 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:52.557914 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ec609-754d6f868c-l66gq"] Oct 30 19:39:52.625598 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:52.625564 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85408ab0-d248-46df-a359-6a213c10fd58-maas-api-tls\") pod \"maas-api-e2e-worker-w4-4ec609-754d6f868c-l66gq\" (UID: \"85408ab0-d248-46df-a359-6a213c10fd58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ec609-754d6f868c-l66gq" Oct 30 19:39:52.625598 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:52.625608 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4r5l\" (UniqueName: \"kubernetes.io/projected/85408ab0-d248-46df-a359-6a213c10fd58-kube-api-access-q4r5l\") pod \"maas-api-e2e-worker-w4-4ec609-754d6f868c-l66gq\" (UID: \"85408ab0-d248-46df-a359-6a213c10fd58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ec609-754d6f868c-l66gq" Oct 30 19:39:52.625868 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:52.625700 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85408ab0-d248-46df-a359-6a213c10fd58-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-4ec609-754d6f868c-l66gq\" (UID: \"85408ab0-d248-46df-a359-6a213c10fd58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ec609-754d6f868c-l66gq" Oct 30 19:39:52.625868 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:39:52.625751 4292 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ec609-serving-cert: secret "maas-api-e2e-worker-w4-4ec609-serving-cert" not found Oct 30 19:39:52.625868 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:52.625786 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85408ab0-d248-46df-a359-6a213c10fd58-metrics-certs\") pod \"maas-api-e2e-worker-w4-4ec609-754d6f868c-l66gq\" (UID: \"85408ab0-d248-46df-a359-6a213c10fd58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ec609-754d6f868c-l66gq" Oct 30 19:39:52.625868 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:39:52.625844 4292 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85408ab0-d248-46df-a359-6a213c10fd58-maas-api-tls podName:85408ab0-d248-46df-a359-6a213c10fd58 nodeName:}" failed. No retries permitted until 2026-10-30 19:39:53.125820523 +0000 UTC m=+896.644135961 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/85408ab0-d248-46df-a359-6a213c10fd58-maas-api-tls") pod "maas-api-e2e-worker-w4-4ec609-754d6f868c-l66gq" (UID: "85408ab0-d248-46df-a359-6a213c10fd58") : secret "maas-api-e2e-worker-w4-4ec609-serving-cert" not found Oct 30 19:39:52.626407 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:52.626385 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85408ab0-d248-46df-a359-6a213c10fd58-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-4ec609-754d6f868c-l66gq\" (UID: \"85408ab0-d248-46df-a359-6a213c10fd58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ec609-754d6f868c-l66gq" Oct 30 19:39:52.628385 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:52.628361 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85408ab0-d248-46df-a359-6a213c10fd58-metrics-certs\") pod \"maas-api-e2e-worker-w4-4ec609-754d6f868c-l66gq\" (UID: \"85408ab0-d248-46df-a359-6a213c10fd58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ec609-754d6f868c-l66gq" Oct 30 19:39:52.684310 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:52.684272 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4r5l\" (UniqueName: \"kubernetes.io/projected/85408ab0-d248-46df-a359-6a213c10fd58-kube-api-access-q4r5l\") pod \"maas-api-e2e-worker-w4-4ec609-754d6f868c-l66gq\" (UID: \"85408ab0-d248-46df-a359-6a213c10fd58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ec609-754d6f868c-l66gq" Oct 30 19:39:53.130112 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:53.130073 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85408ab0-d248-46df-a359-6a213c10fd58-maas-api-tls\") pod \"maas-api-e2e-worker-w4-4ec609-754d6f868c-l66gq\" (UID: \"85408ab0-d248-46df-a359-6a213c10fd58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ec609-754d6f868c-l66gq" Oct 30 19:39:53.132636 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:53.132607 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85408ab0-d248-46df-a359-6a213c10fd58-maas-api-tls\") pod \"maas-api-e2e-worker-w4-4ec609-754d6f868c-l66gq\" (UID: \"85408ab0-d248-46df-a359-6a213c10fd58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ec609-754d6f868c-l66gq" Oct 30 19:39:53.246942 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:53.246906 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-b0a183-5f4f79ff84-pwfs7"] Oct 30 19:39:53.250801 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:53.250778 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-b0a183-5f4f79ff84-pwfs7" Oct 30 19:39:53.303626 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:53.303589 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-b0a183\"" Oct 30 19:39:53.303626 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:53.303619 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-b0a183-dockercfg-4ld9w\"" Oct 30 19:39:53.313246 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:53.313203 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-b0a183-5f4f79ff84-pwfs7"] Oct 30 19:39:53.332113 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:53.332077 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c69db71-a50d-4477-bd35-1eb6c5e47e62-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-b0a183-5f4f79ff84-pwfs7\" (UID: \"7c69db71-a50d-4477-bd35-1eb6c5e47e62\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-b0a183-5f4f79ff84-pwfs7" Oct 30 19:39:53.332279 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:53.332159 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72vbh\" (UniqueName: \"kubernetes.io/projected/7c69db71-a50d-4477-bd35-1eb6c5e47e62-kube-api-access-72vbh\") pod \"payload-processing-e2e-worker-w2-b0a183-5f4f79ff84-pwfs7\" (UID: \"7c69db71-a50d-4477-bd35-1eb6c5e47e62\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-b0a183-5f4f79ff84-pwfs7" Oct 30 19:39:53.371576 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:53.371544 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ec609-754d6f868c-l66gq" Oct 30 19:39:53.433035 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:53.432998 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c69db71-a50d-4477-bd35-1eb6c5e47e62-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-b0a183-5f4f79ff84-pwfs7\" (UID: \"7c69db71-a50d-4477-bd35-1eb6c5e47e62\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-b0a183-5f4f79ff84-pwfs7" Oct 30 19:39:53.433169 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:53.433076 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72vbh\" (UniqueName: \"kubernetes.io/projected/7c69db71-a50d-4477-bd35-1eb6c5e47e62-kube-api-access-72vbh\") pod \"payload-processing-e2e-worker-w2-b0a183-5f4f79ff84-pwfs7\" (UID: \"7c69db71-a50d-4477-bd35-1eb6c5e47e62\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-b0a183-5f4f79ff84-pwfs7" Oct 30 19:39:53.434335 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:53.434309 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c69db71-a50d-4477-bd35-1eb6c5e47e62-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-b0a183-5f4f79ff84-pwfs7\" (UID: \"7c69db71-a50d-4477-bd35-1eb6c5e47e62\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-b0a183-5f4f79ff84-pwfs7" Oct 30 19:39:53.492236 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:53.492201 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72vbh\" (UniqueName: \"kubernetes.io/projected/7c69db71-a50d-4477-bd35-1eb6c5e47e62-kube-api-access-72vbh\") pod \"payload-processing-e2e-worker-w2-b0a183-5f4f79ff84-pwfs7\" (UID: \"7c69db71-a50d-4477-bd35-1eb6c5e47e62\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-b0a183-5f4f79ff84-pwfs7" Oct 30 19:39:53.563813 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:53.563758 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-b0a183-5f4f79ff84-pwfs7" Oct 30 19:39:53.741522 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:53.741494 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ec609-754d6f868c-l66gq"] Oct 30 19:39:53.743323 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:39:53.743294 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85408ab0_d248_46df_a359_6a213c10fd58.slice/crio-fcf2f3ed0eb87f68570b110acb3b6ba91b59f1976b77f75b6d79ddcfdf5b4af6 WatchSource:0}: Error finding container fcf2f3ed0eb87f68570b110acb3b6ba91b59f1976b77f75b6d79ddcfdf5b4af6: Status 404 returned error can't find the container with id fcf2f3ed0eb87f68570b110acb3b6ba91b59f1976b77f75b6d79ddcfdf5b4af6 Oct 30 19:39:53.928375 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:53.928338 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-b0a183-5f4f79ff84-pwfs7"] Oct 30 19:39:53.930255 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:39:53.930192 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c69db71_a50d_4477_bd35_1eb6c5e47e62.slice/crio-34c97823015340820af43737f32dacaec5ae08a2f75e3bfe47c76cb1b09ac341 WatchSource:0}: Error finding container 34c97823015340820af43737f32dacaec5ae08a2f75e3bfe47c76cb1b09ac341: Status 404 returned error can't find the container with id 34c97823015340820af43737f32dacaec5ae08a2f75e3bfe47c76cb1b09ac341 Oct 30 19:39:54.015865 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:54.015765 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-b0a183-5f4f79ff84-pwfs7" event={"ID":"7c69db71-a50d-4477-bd35-1eb6c5e47e62","Type":"ContainerStarted","Data":"5e03571e037122c20cfa35502055b8efec286ae74f4dc4ca7711964ed74df684"} Oct 30 19:39:54.015865 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:54.015837 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-b0a183-5f4f79ff84-pwfs7" event={"ID":"7c69db71-a50d-4477-bd35-1eb6c5e47e62","Type":"ContainerStarted","Data":"34c97823015340820af43737f32dacaec5ae08a2f75e3bfe47c76cb1b09ac341"} Oct 30 19:39:54.016113 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:54.015869 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-b0a183-5f4f79ff84-pwfs7" Oct 30 19:39:54.016918 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:54.016896 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ec609-754d6f868c-l66gq" event={"ID":"85408ab0-d248-46df-a359-6a213c10fd58","Type":"ContainerStarted","Data":"fcf2f3ed0eb87f68570b110acb3b6ba91b59f1976b77f75b6d79ddcfdf5b4af6"} Oct 30 19:39:54.127028 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:54.126976 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-b0a183-5f4f79ff84-pwfs7" podStartSLOduration=1.126963147 podStartE2EDuration="1.126963147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:54.125749491 +0000 UTC m=+897.644064973" watchObservedRunningTime="2026-10-30 19:39:54.126963147 +0000 UTC m=+897.645278605" Oct 30 19:39:55.023121 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:55.023083 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ec609-754d6f868c-l66gq" event={"ID":"85408ab0-d248-46df-a359-6a213c10fd58","Type":"ContainerStarted","Data":"6a851a32d7202183836dae26f5a0ab8070b798da7ea8eb8108aa75760d05350b"} Oct 30 19:39:55.023570 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:55.023336 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ec609-754d6f868c-l66gq" Oct 30 19:39:55.137368 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:55.137304 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ec609-754d6f868c-l66gq" podStartSLOduration=2.292011449 podStartE2EDuration="3.137289352s" totalImagesPullingTime="845.277903ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:55.136606466 +0000 UTC m=+898.654921947" watchObservedRunningTime="2026-10-30 19:39:55.137289352 +0000 UTC m=+898.655604809" Oct 30 19:39:56.926408 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:56.926280 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f214434c-fc7ffd7d6-9wx8r"] Oct 30 19:39:56.926878 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:56.926597 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f214434c-fc7ffd7d6-9wx8r" podUID="cea92f32-a45f-42ee-b0da-79fc9186ec58" containerName="maas-api" containerID="cri-o://bd6fb3586514167093e27ef7eb5de579256f2a444a03bc38be141a0f1a38513e" gracePeriod=30 Oct 30 19:39:57.183787 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:57.183723 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f214434c-fc7ffd7d6-9wx8r" Oct 30 19:39:57.262821 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:57.262784 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cea92f32-a45f-42ee-b0da-79fc9186ec58-kube-api-access-2c6cm\" (UniqueName: \"kubernetes.io/projected/cea92f32-a45f-42ee-b0da-79fc9186ec58-kube-api-access-2c6cm\") pod \"cea92f32-a45f-42ee-b0da-79fc9186ec58\" (UID: \"cea92f32-a45f-42ee-b0da-79fc9186ec58\") " Oct 30 19:39:57.263024 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:57.262858 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cea92f32-a45f-42ee-b0da-79fc9186ec58-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cea92f32-a45f-42ee-b0da-79fc9186ec58-openshift-service-ca\") pod \"cea92f32-a45f-42ee-b0da-79fc9186ec58\" (UID: \"cea92f32-a45f-42ee-b0da-79fc9186ec58\") " Oct 30 19:39:57.263024 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:57.262885 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cea92f32-a45f-42ee-b0da-79fc9186ec58-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cea92f32-a45f-42ee-b0da-79fc9186ec58-maas-api-tls\") pod \"cea92f32-a45f-42ee-b0da-79fc9186ec58\" (UID: \"cea92f32-a45f-42ee-b0da-79fc9186ec58\") " Oct 30 19:39:57.263024 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:57.262903 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cea92f32-a45f-42ee-b0da-79fc9186ec58-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cea92f32-a45f-42ee-b0da-79fc9186ec58-metrics-certs\") pod \"cea92f32-a45f-42ee-b0da-79fc9186ec58\" (UID: \"cea92f32-a45f-42ee-b0da-79fc9186ec58\") " Oct 30 19:39:57.263319 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:57.263291 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cea92f32-a45f-42ee-b0da-79fc9186ec58-openshift-service-ca" pod "cea92f32-a45f-42ee-b0da-79fc9186ec58" (UID: "cea92f32-a45f-42ee-b0da-79fc9186ec58"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:39:57.265077 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:57.265052 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cea92f32-a45f-42ee-b0da-79fc9186ec58-metrics-certs" pod "cea92f32-a45f-42ee-b0da-79fc9186ec58" (UID: "cea92f32-a45f-42ee-b0da-79fc9186ec58"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:39:57.265156 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:57.265136 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cea92f32-a45f-42ee-b0da-79fc9186ec58-kube-api-access-2c6cm" pod "cea92f32-a45f-42ee-b0da-79fc9186ec58" (UID: "cea92f32-a45f-42ee-b0da-79fc9186ec58"). InnerVolumeSpecName "kube-api-access-2c6cm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:39:57.265193 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:57.265144 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cea92f32-a45f-42ee-b0da-79fc9186ec58-maas-api-tls" pod "cea92f32-a45f-42ee-b0da-79fc9186ec58" (UID: "cea92f32-a45f-42ee-b0da-79fc9186ec58"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:39:57.364007 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:57.363965 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c6cm\" (UniqueName: \"kubernetes.io/projected/cea92f32-a45f-42ee-b0da-79fc9186ec58-kube-api-access-2c6cm\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:39:57.364007 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:57.364001 4292 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cea92f32-a45f-42ee-b0da-79fc9186ec58-openshift-service-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:39:57.364007 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:57.364015 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cea92f32-a45f-42ee-b0da-79fc9186ec58-maas-api-tls\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:39:57.364251 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:57.364026 4292 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cea92f32-a45f-42ee-b0da-79fc9186ec58-metrics-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:39:58.036665 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:58.036626 4292 generic.go:356] "Generic (PLEG): container finished" podID="cea92f32-a45f-42ee-b0da-79fc9186ec58" containerID="bd6fb3586514167093e27ef7eb5de579256f2a444a03bc38be141a0f1a38513e" exitCode=0 Oct 30 19:39:58.037482 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:58.036733 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f214434c-fc7ffd7d6-9wx8r" event={"ID":"cea92f32-a45f-42ee-b0da-79fc9186ec58","Type":"ContainerDied","Data":"bd6fb3586514167093e27ef7eb5de579256f2a444a03bc38be141a0f1a38513e"} Oct 30 19:39:58.037482 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:58.036785 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f214434c-fc7ffd7d6-9wx8r" event={"ID":"cea92f32-a45f-42ee-b0da-79fc9186ec58","Type":"ContainerDied","Data":"b1475a72adb564af3aa3fe5a888f778c7c8f96f34952efda2d04e7ef251b73bd"} Oct 30 19:39:58.037482 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:58.036816 4292 scope.go:122] "RemoveContainer" containerID="bd6fb3586514167093e27ef7eb5de579256f2a444a03bc38be141a0f1a38513e" Oct 30 19:39:58.037482 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:58.037131 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f214434c-fc7ffd7d6-9wx8r" Oct 30 19:39:58.048949 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:58.048921 4292 scope.go:122] "RemoveContainer" containerID="bd6fb3586514167093e27ef7eb5de579256f2a444a03bc38be141a0f1a38513e" Oct 30 19:39:58.049329 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:39:58.049306 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd6fb3586514167093e27ef7eb5de579256f2a444a03bc38be141a0f1a38513e\": container with ID starting with bd6fb3586514167093e27ef7eb5de579256f2a444a03bc38be141a0f1a38513e not found: ID does not exist" containerID="bd6fb3586514167093e27ef7eb5de579256f2a444a03bc38be141a0f1a38513e" Oct 30 19:39:58.049450 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:58.049338 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd6fb3586514167093e27ef7eb5de579256f2a444a03bc38be141a0f1a38513e"} err="failed to get container status \"bd6fb3586514167093e27ef7eb5de579256f2a444a03bc38be141a0f1a38513e\": rpc error: code = NotFound desc = could not find container \"bd6fb3586514167093e27ef7eb5de579256f2a444a03bc38be141a0f1a38513e\": container with ID starting with bd6fb3586514167093e27ef7eb5de579256f2a444a03bc38be141a0f1a38513e not found: ID does not exist" Oct 30 19:39:58.202044 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:58.201269 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f214434c-fc7ffd7d6-9wx8r"] Oct 30 19:39:58.241705 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:58.241664 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f214434c-fc7ffd7d6-9wx8r"] Oct 30 19:39:58.580758 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:39:58.580728 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cea92f32-a45f-42ee-b0da-79fc9186ec58" path="/var/lib/kubelet/pods/cea92f32-a45f-42ee-b0da-79fc9186ec58/volumes" Oct 30 19:40:01.032734 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:01.032704 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ec609-754d6f868c-l66gq" Oct 30 19:40:05.033167 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:05.033137 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-b0a183-5f4f79ff84-pwfs7" Oct 30 19:40:08.970833 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:08.970790 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-bbfa07-658b9f6ff5-qxnqd"] Oct 30 19:40:08.971308 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:08.971264 4292 cpu_manager.go:414] "containerMap: removing container" podUID="cea92f32-a45f-42ee-b0da-79fc9186ec58" containerName="maas-api" Oct 30 19:40:08.971308 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:08.971282 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cea92f32-a45f-42ee-b0da-79fc9186ec58" containerName="maas-api" Oct 30 19:40:08.971469 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:08.971406 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="cea92f32-a45f-42ee-b0da-79fc9186ec58" containerName="maas-api" Oct 30 19:40:08.975873 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:08.975848 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bbfa07-658b9f6ff5-qxnqd" Oct 30 19:40:09.028748 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:09.028714 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-bbfa07-658b9f6ff5-qxnqd"] Oct 30 19:40:09.073896 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:09.073852 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-bbfa07-dockercfg-c269m\"" Oct 30 19:40:09.073896 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:09.073852 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-bbfa07\"" Oct 30 19:40:09.076197 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:09.076160 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhndq\" (UniqueName: \"kubernetes.io/projected/d0946fd9-55cf-4c53-90af-22020e7b41ac-kube-api-access-qhndq\") pod \"payload-pre-processing-e2e-worker-w3-bbfa07-658b9f6ff5-qxnqd\" (UID: \"d0946fd9-55cf-4c53-90af-22020e7b41ac\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bbfa07-658b9f6ff5-qxnqd" Oct 30 19:40:09.076336 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:09.076265 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0946fd9-55cf-4c53-90af-22020e7b41ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-bbfa07-658b9f6ff5-qxnqd\" (UID: \"d0946fd9-55cf-4c53-90af-22020e7b41ac\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bbfa07-658b9f6ff5-qxnqd" Oct 30 19:40:09.151988 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:09.151954 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-bbfa07-5d784fb568-6ddcf"] Oct 30 19:40:09.155472 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:09.155447 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-bbfa07-5d784fb568-6ddcf" Oct 30 19:40:09.177750 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:09.177705 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhndq\" (UniqueName: \"kubernetes.io/projected/d0946fd9-55cf-4c53-90af-22020e7b41ac-kube-api-access-qhndq\") pod \"payload-pre-processing-e2e-worker-w3-bbfa07-658b9f6ff5-qxnqd\" (UID: \"d0946fd9-55cf-4c53-90af-22020e7b41ac\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bbfa07-658b9f6ff5-qxnqd" Oct 30 19:40:09.177932 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:09.177761 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0946fd9-55cf-4c53-90af-22020e7b41ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-bbfa07-658b9f6ff5-qxnqd\" (UID: \"d0946fd9-55cf-4c53-90af-22020e7b41ac\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bbfa07-658b9f6ff5-qxnqd" Oct 30 19:40:09.178748 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:09.178686 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0946fd9-55cf-4c53-90af-22020e7b41ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-bbfa07-658b9f6ff5-qxnqd\" (UID: \"d0946fd9-55cf-4c53-90af-22020e7b41ac\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bbfa07-658b9f6ff5-qxnqd" Oct 30 19:40:09.235336 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:09.235238 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-bbfa07-5d784fb568-6ddcf"] Oct 30 19:40:09.238260 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:09.238231 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhndq\" (UniqueName: \"kubernetes.io/projected/d0946fd9-55cf-4c53-90af-22020e7b41ac-kube-api-access-qhndq\") pod \"payload-pre-processing-e2e-worker-w3-bbfa07-658b9f6ff5-qxnqd\" (UID: \"d0946fd9-55cf-4c53-90af-22020e7b41ac\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bbfa07-658b9f6ff5-qxnqd" Oct 30 19:40:09.278829 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:09.278795 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq7ds\" (UniqueName: \"kubernetes.io/projected/1f356d7e-7b1a-4b51-a36f-a809a4a4f91f-kube-api-access-lq7ds\") pod \"payload-processing-e2e-worker-w3-bbfa07-5d784fb568-6ddcf\" (UID: \"1f356d7e-7b1a-4b51-a36f-a809a4a4f91f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-bbfa07-5d784fb568-6ddcf" Oct 30 19:40:09.279013 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:09.278836 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f356d7e-7b1a-4b51-a36f-a809a4a4f91f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-bbfa07-5d784fb568-6ddcf\" (UID: \"1f356d7e-7b1a-4b51-a36f-a809a4a4f91f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-bbfa07-5d784fb568-6ddcf" Oct 30 19:40:09.289785 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:09.289755 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bbfa07-658b9f6ff5-qxnqd" Oct 30 19:40:09.379701 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:09.379658 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq7ds\" (UniqueName: \"kubernetes.io/projected/1f356d7e-7b1a-4b51-a36f-a809a4a4f91f-kube-api-access-lq7ds\") pod \"payload-processing-e2e-worker-w3-bbfa07-5d784fb568-6ddcf\" (UID: \"1f356d7e-7b1a-4b51-a36f-a809a4a4f91f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-bbfa07-5d784fb568-6ddcf" Oct 30 19:40:09.379887 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:09.379713 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f356d7e-7b1a-4b51-a36f-a809a4a4f91f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-bbfa07-5d784fb568-6ddcf\" (UID: \"1f356d7e-7b1a-4b51-a36f-a809a4a4f91f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-bbfa07-5d784fb568-6ddcf" Oct 30 19:40:09.380508 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:09.380483 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f356d7e-7b1a-4b51-a36f-a809a4a4f91f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-bbfa07-5d784fb568-6ddcf\" (UID: \"1f356d7e-7b1a-4b51-a36f-a809a4a4f91f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-bbfa07-5d784fb568-6ddcf" Oct 30 19:40:09.438310 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:09.438276 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq7ds\" (UniqueName: \"kubernetes.io/projected/1f356d7e-7b1a-4b51-a36f-a809a4a4f91f-kube-api-access-lq7ds\") pod \"payload-processing-e2e-worker-w3-bbfa07-5d784fb568-6ddcf\" (UID: \"1f356d7e-7b1a-4b51-a36f-a809a4a4f91f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-bbfa07-5d784fb568-6ddcf" Oct 30 19:40:09.470934 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:09.470890 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-bbfa07-5d784fb568-6ddcf" Oct 30 19:40:09.658931 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:09.658907 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-bbfa07-658b9f6ff5-qxnqd"] Oct 30 19:40:09.661057 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:40:09.661026 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0946fd9_55cf_4c53_90af_22020e7b41ac.slice/crio-d1edd8624ed35416998c8c8a2caa045d1e6b3936b9859befe11cfce3b687678b WatchSource:0}: Error finding container d1edd8624ed35416998c8c8a2caa045d1e6b3936b9859befe11cfce3b687678b: Status 404 returned error can't find the container with id d1edd8624ed35416998c8c8a2caa045d1e6b3936b9859befe11cfce3b687678b Oct 30 19:40:09.838667 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:09.838628 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-bbfa07-5d784fb568-6ddcf"] Oct 30 19:40:09.840884 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:40:09.840853 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f356d7e_7b1a_4b51_a36f_a809a4a4f91f.slice/crio-e5cba0945f72777adc360d587d966c130dbd755103e38a853715a2052379339b WatchSource:0}: Error finding container e5cba0945f72777adc360d587d966c130dbd755103e38a853715a2052379339b: Status 404 returned error can't find the container with id e5cba0945f72777adc360d587d966c130dbd755103e38a853715a2052379339b Oct 30 19:40:10.093595 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:10.093487 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-bbfa07-5d784fb568-6ddcf" event={"ID":"1f356d7e-7b1a-4b51-a36f-a809a4a4f91f","Type":"ContainerStarted","Data":"2cffd75e0dbf07dd8029601824695fe3f485818d6b7e0f463051d032b36930e7"} Oct 30 19:40:10.094029 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:10.093600 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-bbfa07-5d784fb568-6ddcf" Oct 30 19:40:10.094029 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:10.093630 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-bbfa07-5d784fb568-6ddcf" event={"ID":"1f356d7e-7b1a-4b51-a36f-a809a4a4f91f","Type":"ContainerStarted","Data":"e5cba0945f72777adc360d587d966c130dbd755103e38a853715a2052379339b"} Oct 30 19:40:10.095066 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:10.095037 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bbfa07-658b9f6ff5-qxnqd" event={"ID":"d0946fd9-55cf-4c53-90af-22020e7b41ac","Type":"ContainerStarted","Data":"ca77cb60489d7c4ab5a0997b1863d00a80ca659ce6d7f37b9a877f6295747064"} Oct 30 19:40:10.095178 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:10.095089 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bbfa07-658b9f6ff5-qxnqd" event={"ID":"d0946fd9-55cf-4c53-90af-22020e7b41ac","Type":"ContainerStarted","Data":"d1edd8624ed35416998c8c8a2caa045d1e6b3936b9859befe11cfce3b687678b"} Oct 30 19:40:10.095178 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:10.095133 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bbfa07-658b9f6ff5-qxnqd" Oct 30 19:40:10.211383 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:10.211300 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-bbfa07-5d784fb568-6ddcf" podStartSLOduration=1.2112800400000001 podStartE2EDuration="1.21128004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:10.209535815 +0000 UTC m=+913.727851285" watchObservedRunningTime="2026-10-30 19:40:10.21128004 +0000 UTC m=+913.729595498" Oct 30 19:40:10.324306 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:10.324238 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bbfa07-658b9f6ff5-qxnqd" podStartSLOduration=2.324217167 podStartE2EDuration="2.324217167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:10.322923612 +0000 UTC m=+913.841239095" watchObservedRunningTime="2026-10-30 19:40:10.324217167 +0000 UTC m=+913.842532626" Oct 30 19:40:13.221480 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:40:13.221441 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:40:18.958273 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:18.958236 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-98a19b-779f877d8-hgncx"] Oct 30 19:40:18.961896 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:18.961871 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-98a19b-779f877d8-hgncx" Oct 30 19:40:19.058162 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:19.058130 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-98a19b-dockercfg-pnfkq\"" Oct 30 19:40:19.058162 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:19.058165 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-98a19b\"" Oct 30 19:40:19.059682 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:19.059657 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-98a19b-779f877d8-hgncx"] Oct 30 19:40:19.071040 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:19.071008 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbj5g\" (UniqueName: \"kubernetes.io/projected/e8810ca2-97ba-4a61-84da-fc464be80c17-kube-api-access-rbj5g\") pod \"payload-processing-e2e-worker-w1-98a19b-779f877d8-hgncx\" (UID: \"e8810ca2-97ba-4a61-84da-fc464be80c17\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-98a19b-779f877d8-hgncx" Oct 30 19:40:19.071174 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:19.071113 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8810ca2-97ba-4a61-84da-fc464be80c17-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-98a19b-779f877d8-hgncx\" (UID: \"e8810ca2-97ba-4a61-84da-fc464be80c17\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-98a19b-779f877d8-hgncx" Oct 30 19:40:19.172409 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:19.172376 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8810ca2-97ba-4a61-84da-fc464be80c17-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-98a19b-779f877d8-hgncx\" (UID: \"e8810ca2-97ba-4a61-84da-fc464be80c17\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-98a19b-779f877d8-hgncx" Oct 30 19:40:19.172565 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:19.172447 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbj5g\" (UniqueName: \"kubernetes.io/projected/e8810ca2-97ba-4a61-84da-fc464be80c17-kube-api-access-rbj5g\") pod \"payload-processing-e2e-worker-w1-98a19b-779f877d8-hgncx\" (UID: \"e8810ca2-97ba-4a61-84da-fc464be80c17\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-98a19b-779f877d8-hgncx" Oct 30 19:40:19.173008 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:19.172990 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8810ca2-97ba-4a61-84da-fc464be80c17-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-98a19b-779f877d8-hgncx\" (UID: \"e8810ca2-97ba-4a61-84da-fc464be80c17\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-98a19b-779f877d8-hgncx" Oct 30 19:40:19.231325 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:19.231258 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbj5g\" (UniqueName: \"kubernetes.io/projected/e8810ca2-97ba-4a61-84da-fc464be80c17-kube-api-access-rbj5g\") pod \"payload-processing-e2e-worker-w1-98a19b-779f877d8-hgncx\" (UID: \"e8810ca2-97ba-4a61-84da-fc464be80c17\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-98a19b-779f877d8-hgncx" Oct 30 19:40:19.274888 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:19.274857 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-98a19b-779f877d8-hgncx" Oct 30 19:40:19.663169 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:19.663144 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-98a19b-779f877d8-hgncx"] Oct 30 19:40:19.664108 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:40:19.664080 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8810ca2_97ba_4a61_84da_fc464be80c17.slice/crio-854a73591375fbffeb5d5fddb2c9a6c00470b8671a065c7af34007d86fde8fde WatchSource:0}: Error finding container 854a73591375fbffeb5d5fddb2c9a6c00470b8671a065c7af34007d86fde8fde: Status 404 returned error can't find the container with id 854a73591375fbffeb5d5fddb2c9a6c00470b8671a065c7af34007d86fde8fde Oct 30 19:40:20.138150 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:20.138120 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-98a19b-779f877d8-hgncx" event={"ID":"e8810ca2-97ba-4a61-84da-fc464be80c17","Type":"ContainerStarted","Data":"bfb441cca7276ae3e08df2c77f3f4ec1a6c0a473e4347a1eec4ccee3f07a13b1"} Oct 30 19:40:20.138589 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:20.138167 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-98a19b-779f877d8-hgncx" event={"ID":"e8810ca2-97ba-4a61-84da-fc464be80c17","Type":"ContainerStarted","Data":"854a73591375fbffeb5d5fddb2c9a6c00470b8671a065c7af34007d86fde8fde"} Oct 30 19:40:20.138589 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:20.138198 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-98a19b-779f877d8-hgncx" Oct 30 19:40:20.250523 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:20.250432 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-98a19b-779f877d8-hgncx" podStartSLOduration=2.2504103 podStartE2EDuration="2.2504103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:20.248530323 +0000 UTC m=+923.766845823" watchObservedRunningTime="2026-10-30 19:40:20.2504103 +0000 UTC m=+923.768725759" Oct 30 19:40:21.102560 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:21.102534 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bbfa07-658b9f6ff5-qxnqd" Oct 30 19:40:21.102741 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:21.102598 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-bbfa07-5d784fb568-6ddcf" Oct 30 19:40:31.146496 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:31.146459 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-98a19b-779f877d8-hgncx" Oct 30 19:40:34.907600 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:34.907569 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-98a19b-9477d9fdc-b2sls"] Oct 30 19:40:34.912287 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:34.912265 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-98a19b-9477d9fdc-b2sls" Oct 30 19:40:34.961695 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:34.961661 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-98a19b-serving-cert\"" Oct 30 19:40:35.010115 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:35.010079 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-98a19b-9477d9fdc-b2sls"] Oct 30 19:40:35.014136 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:35.014117 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfb77\" (UniqueName: \"kubernetes.io/projected/f7d43882-57a4-43f7-9ad9-c5174007f344-kube-api-access-bfb77\") pod \"maas-api-e2e-worker-w1-98a19b-9477d9fdc-b2sls\" (UID: \"f7d43882-57a4-43f7-9ad9-c5174007f344\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-98a19b-9477d9fdc-b2sls" Oct 30 19:40:35.014211 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:35.014165 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7d43882-57a4-43f7-9ad9-c5174007f344-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-98a19b-9477d9fdc-b2sls\" (UID: \"f7d43882-57a4-43f7-9ad9-c5174007f344\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-98a19b-9477d9fdc-b2sls" Oct 30 19:40:35.014318 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:35.014298 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7d43882-57a4-43f7-9ad9-c5174007f344-maas-api-tls\") pod \"maas-api-e2e-worker-w1-98a19b-9477d9fdc-b2sls\" (UID: \"f7d43882-57a4-43f7-9ad9-c5174007f344\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-98a19b-9477d9fdc-b2sls" Oct 30 19:40:35.014385 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:35.014325 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7d43882-57a4-43f7-9ad9-c5174007f344-metrics-certs\") pod \"maas-api-e2e-worker-w1-98a19b-9477d9fdc-b2sls\" (UID: \"f7d43882-57a4-43f7-9ad9-c5174007f344\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-98a19b-9477d9fdc-b2sls" Oct 30 19:40:35.115021 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:35.114980 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7d43882-57a4-43f7-9ad9-c5174007f344-maas-api-tls\") pod \"maas-api-e2e-worker-w1-98a19b-9477d9fdc-b2sls\" (UID: \"f7d43882-57a4-43f7-9ad9-c5174007f344\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-98a19b-9477d9fdc-b2sls" Oct 30 19:40:35.115021 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:35.115012 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7d43882-57a4-43f7-9ad9-c5174007f344-metrics-certs\") pod \"maas-api-e2e-worker-w1-98a19b-9477d9fdc-b2sls\" (UID: \"f7d43882-57a4-43f7-9ad9-c5174007f344\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-98a19b-9477d9fdc-b2sls" Oct 30 19:40:35.115021 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:35.115032 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfb77\" (UniqueName: \"kubernetes.io/projected/f7d43882-57a4-43f7-9ad9-c5174007f344-kube-api-access-bfb77\") pod \"maas-api-e2e-worker-w1-98a19b-9477d9fdc-b2sls\" (UID: \"f7d43882-57a4-43f7-9ad9-c5174007f344\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-98a19b-9477d9fdc-b2sls" Oct 30 19:40:35.115288 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:35.115076 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7d43882-57a4-43f7-9ad9-c5174007f344-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-98a19b-9477d9fdc-b2sls\" (UID: \"f7d43882-57a4-43f7-9ad9-c5174007f344\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-98a19b-9477d9fdc-b2sls" Oct 30 19:40:35.115288 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:40:35.115096 4292 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-98a19b-serving-cert: secret "maas-api-e2e-worker-w1-98a19b-serving-cert" not found Oct 30 19:40:35.115288 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:40:35.115173 4292 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7d43882-57a4-43f7-9ad9-c5174007f344-maas-api-tls podName:f7d43882-57a4-43f7-9ad9-c5174007f344 nodeName:}" failed. No retries permitted until 2026-10-30 19:40:35.615151676 +0000 UTC m=+939.133467130 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f7d43882-57a4-43f7-9ad9-c5174007f344-maas-api-tls") pod "maas-api-e2e-worker-w1-98a19b-9477d9fdc-b2sls" (UID: "f7d43882-57a4-43f7-9ad9-c5174007f344") : secret "maas-api-e2e-worker-w1-98a19b-serving-cert" not found Oct 30 19:40:35.115781 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:35.115764 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7d43882-57a4-43f7-9ad9-c5174007f344-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-98a19b-9477d9fdc-b2sls\" (UID: \"f7d43882-57a4-43f7-9ad9-c5174007f344\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-98a19b-9477d9fdc-b2sls" Oct 30 19:40:35.117580 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:35.117560 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7d43882-57a4-43f7-9ad9-c5174007f344-metrics-certs\") pod \"maas-api-e2e-worker-w1-98a19b-9477d9fdc-b2sls\" (UID: \"f7d43882-57a4-43f7-9ad9-c5174007f344\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-98a19b-9477d9fdc-b2sls" Oct 30 19:40:35.173972 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:35.173891 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfb77\" (UniqueName: \"kubernetes.io/projected/f7d43882-57a4-43f7-9ad9-c5174007f344-kube-api-access-bfb77\") pod \"maas-api-e2e-worker-w1-98a19b-9477d9fdc-b2sls\" (UID: \"f7d43882-57a4-43f7-9ad9-c5174007f344\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-98a19b-9477d9fdc-b2sls" Oct 30 19:40:35.619492 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:35.619453 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7d43882-57a4-43f7-9ad9-c5174007f344-maas-api-tls\") pod \"maas-api-e2e-worker-w1-98a19b-9477d9fdc-b2sls\" (UID: \"f7d43882-57a4-43f7-9ad9-c5174007f344\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-98a19b-9477d9fdc-b2sls" Oct 30 19:40:35.634840 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:35.634805 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7d43882-57a4-43f7-9ad9-c5174007f344-maas-api-tls\") pod \"maas-api-e2e-worker-w1-98a19b-9477d9fdc-b2sls\" (UID: \"f7d43882-57a4-43f7-9ad9-c5174007f344\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-98a19b-9477d9fdc-b2sls" Oct 30 19:40:35.826449 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:35.826402 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-98a19b-9477d9fdc-b2sls" Oct 30 19:40:36.191736 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:36.191564 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-98a19b-9477d9fdc-b2sls"] Oct 30 19:40:36.194154 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:40:36.194127 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7d43882_57a4_43f7_9ad9_c5174007f344.slice/crio-86fd8b7688c4c9c08e762238036d1a43cf181b8343ff57a0dcb0ddc56387b50d WatchSource:0}: Error finding container 86fd8b7688c4c9c08e762238036d1a43cf181b8343ff57a0dcb0ddc56387b50d: Status 404 returned error can't find the container with id 86fd8b7688c4c9c08e762238036d1a43cf181b8343ff57a0dcb0ddc56387b50d Oct 30 19:40:36.206976 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:36.206946 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-98a19b-9477d9fdc-b2sls" event={"ID":"f7d43882-57a4-43f7-9ad9-c5174007f344","Type":"ContainerStarted","Data":"86fd8b7688c4c9c08e762238036d1a43cf181b8343ff57a0dcb0ddc56387b50d"} Oct 30 19:40:37.213314 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:37.213264 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-98a19b-9477d9fdc-b2sls" event={"ID":"f7d43882-57a4-43f7-9ad9-c5174007f344","Type":"ContainerStarted","Data":"b01f2cfbc155b30f4c01717cbdabc03df2223ce2e9b6a338ac525bbc06e43052"} Oct 30 19:40:37.213768 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:37.213336 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-98a19b-9477d9fdc-b2sls" Oct 30 19:40:37.355160 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:37.355057 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-98a19b-9477d9fdc-b2sls" podStartSLOduration=2.4671216879999998 podStartE2EDuration="3.355040616s" totalImagesPullingTime="887.918928ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:37.35361543 +0000 UTC m=+940.871930888" watchObservedRunningTime="2026-10-30 19:40:37.355040616 +0000 UTC m=+940.873356073" Oct 30 19:40:43.220404 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:40:43.220301 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:40:43.223901 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:43.223878 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-98a19b-9477d9fdc-b2sls" Oct 30 19:40:45.933142 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:45.933101 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x"] Oct 30 19:40:45.936670 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:45.936645 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x" Oct 30 19:40:45.986423 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:45.986378 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-bbfa07-openshift-default-dockercfg-vj7kd\"" Oct 30 19:40:46.018001 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:46.017962 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/931c961c-cdf7-442e-8a6f-ff766e813cad-istio-envoy\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x\" (UID: \"931c961c-cdf7-442e-8a6f-ff766e813cad\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x" Oct 30 19:40:46.018193 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:46.018020 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/931c961c-cdf7-442e-8a6f-ff766e813cad-workload-certs\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x\" (UID: \"931c961c-cdf7-442e-8a6f-ff766e813cad\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x" Oct 30 19:40:46.018193 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:46.018119 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/931c961c-cdf7-442e-8a6f-ff766e813cad-istio-data\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x\" (UID: \"931c961c-cdf7-442e-8a6f-ff766e813cad\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x" Oct 30 19:40:46.018327 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:46.018190 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/931c961c-cdf7-442e-8a6f-ff766e813cad-credential-socket\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x\" (UID: \"931c961c-cdf7-442e-8a6f-ff766e813cad\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x" Oct 30 19:40:46.018327 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:46.018228 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/931c961c-cdf7-442e-8a6f-ff766e813cad-istio-podinfo\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x\" (UID: \"931c961c-cdf7-442e-8a6f-ff766e813cad\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x" Oct 30 19:40:46.018327 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:46.018256 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh6kg\" (UniqueName: \"kubernetes.io/projected/931c961c-cdf7-442e-8a6f-ff766e813cad-kube-api-access-hh6kg\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x\" (UID: \"931c961c-cdf7-442e-8a6f-ff766e813cad\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x" Oct 30 19:40:46.018506 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:46.018335 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/931c961c-cdf7-442e-8a6f-ff766e813cad-istio-token\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x\" (UID: \"931c961c-cdf7-442e-8a6f-ff766e813cad\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x" Oct 30 19:40:46.018506 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:46.018435 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/931c961c-cdf7-442e-8a6f-ff766e813cad-workload-socket\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x\" (UID: \"931c961c-cdf7-442e-8a6f-ff766e813cad\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x" Oct 30 19:40:46.018506 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:46.018461 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/931c961c-cdf7-442e-8a6f-ff766e813cad-istiod-ca-cert\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x\" (UID: \"931c961c-cdf7-442e-8a6f-ff766e813cad\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x" Oct 30 19:40:46.119102 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:46.119065 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/931c961c-cdf7-442e-8a6f-ff766e813cad-istio-data\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x\" (UID: \"931c961c-cdf7-442e-8a6f-ff766e813cad\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x" Oct 30 19:40:46.119301 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:46.119117 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/931c961c-cdf7-442e-8a6f-ff766e813cad-credential-socket\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x\" (UID: \"931c961c-cdf7-442e-8a6f-ff766e813cad\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x" Oct 30 19:40:46.119301 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:46.119137 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/931c961c-cdf7-442e-8a6f-ff766e813cad-istio-podinfo\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x\" (UID: \"931c961c-cdf7-442e-8a6f-ff766e813cad\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x" Oct 30 19:40:46.119301 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:46.119156 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh6kg\" (UniqueName: \"kubernetes.io/projected/931c961c-cdf7-442e-8a6f-ff766e813cad-kube-api-access-hh6kg\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x\" (UID: \"931c961c-cdf7-442e-8a6f-ff766e813cad\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x" Oct 30 19:40:46.119301 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:46.119176 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/931c961c-cdf7-442e-8a6f-ff766e813cad-istio-token\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x\" (UID: \"931c961c-cdf7-442e-8a6f-ff766e813cad\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x" Oct 30 19:40:46.119301 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:46.119213 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/931c961c-cdf7-442e-8a6f-ff766e813cad-workload-socket\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x\" (UID: \"931c961c-cdf7-442e-8a6f-ff766e813cad\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x" Oct 30 19:40:46.119301 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:46.119233 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/931c961c-cdf7-442e-8a6f-ff766e813cad-istiod-ca-cert\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x\" (UID: \"931c961c-cdf7-442e-8a6f-ff766e813cad\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x" Oct 30 19:40:46.119301 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:46.119266 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/931c961c-cdf7-442e-8a6f-ff766e813cad-istio-envoy\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x\" (UID: \"931c961c-cdf7-442e-8a6f-ff766e813cad\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x" Oct 30 19:40:46.119301 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:46.119304 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/931c961c-cdf7-442e-8a6f-ff766e813cad-workload-certs\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x\" (UID: \"931c961c-cdf7-442e-8a6f-ff766e813cad\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x" Oct 30 19:40:46.119721 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:46.119612 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/931c961c-cdf7-442e-8a6f-ff766e813cad-istio-data\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x\" (UID: \"931c961c-cdf7-442e-8a6f-ff766e813cad\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x" Oct 30 19:40:46.119721 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:46.119645 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/931c961c-cdf7-442e-8a6f-ff766e813cad-credential-socket\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x\" (UID: \"931c961c-cdf7-442e-8a6f-ff766e813cad\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x" Oct 30 19:40:46.119721 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:46.119688 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/931c961c-cdf7-442e-8a6f-ff766e813cad-workload-certs\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x\" (UID: \"931c961c-cdf7-442e-8a6f-ff766e813cad\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x" Oct 30 19:40:46.120072 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:46.120046 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/931c961c-cdf7-442e-8a6f-ff766e813cad-istiod-ca-cert\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x\" (UID: \"931c961c-cdf7-442e-8a6f-ff766e813cad\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x" Oct 30 19:40:46.120275 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:46.120257 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/931c961c-cdf7-442e-8a6f-ff766e813cad-workload-socket\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x\" (UID: \"931c961c-cdf7-442e-8a6f-ff766e813cad\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x" Oct 30 19:40:46.121641 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:46.121620 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/931c961c-cdf7-442e-8a6f-ff766e813cad-istio-envoy\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x\" (UID: \"931c961c-cdf7-442e-8a6f-ff766e813cad\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x" Oct 30 19:40:46.121891 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:46.121869 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/931c961c-cdf7-442e-8a6f-ff766e813cad-istio-podinfo\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x\" (UID: \"931c961c-cdf7-442e-8a6f-ff766e813cad\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x" Oct 30 19:40:46.139236 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:46.139208 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x"] Oct 30 19:40:46.177933 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:46.177904 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/931c961c-cdf7-442e-8a6f-ff766e813cad-istio-token\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x\" (UID: \"931c961c-cdf7-442e-8a6f-ff766e813cad\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x" Oct 30 19:40:46.185380 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:46.185300 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh6kg\" (UniqueName: \"kubernetes.io/projected/931c961c-cdf7-442e-8a6f-ff766e813cad-kube-api-access-hh6kg\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x\" (UID: \"931c961c-cdf7-442e-8a6f-ff766e813cad\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x" Oct 30 19:40:46.249397 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:46.249359 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x" Oct 30 19:40:46.668181 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:46.668155 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x"] Oct 30 19:40:46.670481 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:40:46.670451 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod931c961c_cdf7_442e_8a6f_ff766e813cad.slice/crio-79e89fbd3ac0387c8bd20f2af185da9b88697b1b01e0622e07aef562acad45fe WatchSource:0}: Error finding container 79e89fbd3ac0387c8bd20f2af185da9b88697b1b01e0622e07aef562acad45fe: Status 404 returned error can't find the container with id 79e89fbd3ac0387c8bd20f2af185da9b88697b1b01e0622e07aef562acad45fe Oct 30 19:40:46.672755 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:46.672722 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:40:46.672825 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:46.672789 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:40:46.672825 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:46.672821 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:40:47.276317 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:47.276281 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x" event={"ID":"931c961c-cdf7-442e-8a6f-ff766e813cad","Type":"ContainerStarted","Data":"8a7ea8f7f609e26bbb5bc4554256a089c7cc68afae431b9ffff3129e79456c20"} Oct 30 19:40:47.276757 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:47.276342 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x" event={"ID":"931c961c-cdf7-442e-8a6f-ff766e813cad","Type":"ContainerStarted","Data":"79e89fbd3ac0387c8bd20f2af185da9b88697b1b01e0622e07aef562acad45fe"} Oct 30 19:40:47.465377 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:47.456871 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x" podStartSLOduration=2.456849208 podStartE2EDuration="2.456849208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:47.454913443 +0000 UTC m=+950.973228926" watchObservedRunningTime="2026-10-30 19:40:47.456849208 +0000 UTC m=+950.975164665" Oct 30 19:40:48.265409 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:48.265372 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x" Oct 30 19:40:48.265792 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:48.265765 4292 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 19:40:48.265975 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:48.265950 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x" podUID="931c961c-cdf7-442e-8a6f-ff766e813cad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 19:40:49.253881 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:49.253837 4292 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 19:40:49.254533 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:49.254498 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x" podUID="931c961c-cdf7-442e-8a6f-ff766e813cad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 19:40:50.254400 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:50.253809 4292 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 19:40:50.254400 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:50.253894 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x" podUID="931c961c-cdf7-442e-8a6f-ff766e813cad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 19:40:51.266305 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:51.265824 4292 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 19:40:51.266305 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:51.265889 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x" podUID="931c961c-cdf7-442e-8a6f-ff766e813cad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 19:40:52.258378 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:52.256812 4292 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 19:40:52.258378 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:52.256883 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x" podUID="931c961c-cdf7-442e-8a6f-ff766e813cad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 19:40:53.254999 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:53.254962 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x" Oct 30 19:40:53.255468 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:53.255314 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x" Oct 30 19:40:53.256198 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:40:53.256176 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x" Oct 30 19:41:13.123746 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:13.123709 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-a74d15/premium-w0-a74d15-kserve-864f45fdfb-949gm"] Oct 30 19:41:13.128605 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:13.128584 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a74d15/premium-w0-a74d15-kserve-864f45fdfb-949gm" Oct 30 19:41:13.198686 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:13.198651 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb0ced80-4c4f-41dc-883c-b1a21e0f11f8-dshm\") pod \"premium-w0-a74d15-kserve-864f45fdfb-949gm\" (UID: \"eb0ced80-4c4f-41dc-883c-b1a21e0f11f8\") " pod="e2e-models-e2e-worker-w0-a74d15/premium-w0-a74d15-kserve-864f45fdfb-949gm" Oct 30 19:41:13.198686 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:13.198686 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb0ced80-4c4f-41dc-883c-b1a21e0f11f8-tmp-dir\") pod \"premium-w0-a74d15-kserve-864f45fdfb-949gm\" (UID: \"eb0ced80-4c4f-41dc-883c-b1a21e0f11f8\") " pod="e2e-models-e2e-worker-w0-a74d15/premium-w0-a74d15-kserve-864f45fdfb-949gm" Oct 30 19:41:13.198938 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:13.198710 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb0ced80-4c4f-41dc-883c-b1a21e0f11f8-tls-certs\") pod \"premium-w0-a74d15-kserve-864f45fdfb-949gm\" (UID: \"eb0ced80-4c4f-41dc-883c-b1a21e0f11f8\") " pod="e2e-models-e2e-worker-w0-a74d15/premium-w0-a74d15-kserve-864f45fdfb-949gm" Oct 30 19:41:13.198938 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:13.198910 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb0ced80-4c4f-41dc-883c-b1a21e0f11f8-model-cache\") pod \"premium-w0-a74d15-kserve-864f45fdfb-949gm\" (UID: \"eb0ced80-4c4f-41dc-883c-b1a21e0f11f8\") " pod="e2e-models-e2e-worker-w0-a74d15/premium-w0-a74d15-kserve-864f45fdfb-949gm" Oct 30 19:41:13.199049 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:13.198961 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv7f4\" (UniqueName: \"kubernetes.io/projected/eb0ced80-4c4f-41dc-883c-b1a21e0f11f8-kube-api-access-zv7f4\") pod \"premium-w0-a74d15-kserve-864f45fdfb-949gm\" (UID: \"eb0ced80-4c4f-41dc-883c-b1a21e0f11f8\") " pod="e2e-models-e2e-worker-w0-a74d15/premium-w0-a74d15-kserve-864f45fdfb-949gm" Oct 30 19:41:13.199049 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:13.199013 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb0ced80-4c4f-41dc-883c-b1a21e0f11f8-home\") pod \"premium-w0-a74d15-kserve-864f45fdfb-949gm\" (UID: \"eb0ced80-4c4f-41dc-883c-b1a21e0f11f8\") " pod="e2e-models-e2e-worker-w0-a74d15/premium-w0-a74d15-kserve-864f45fdfb-949gm" Oct 30 19:41:13.220562 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:41:13.220523 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:41:13.225520 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:13.225474 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a74d15\"/\"premium-w0-a74d15-kserve-self-signed-certs\"" Oct 30 19:41:13.225705 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:13.225546 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-a74d15\"/\"kube-root-ca.crt\"" Oct 30 19:41:13.225705 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:13.225573 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a74d15\"/\"default-dockercfg-4fxps\"" Oct 30 19:41:13.225705 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:13.225631 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-a74d15\"/\"openshift-service-ca.crt\"" Oct 30 19:41:13.227206 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:13.227188 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a74d15/premium-w0-a74d15-kserve-864f45fdfb-949gm"] Oct 30 19:41:13.300031 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:13.299987 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb0ced80-4c4f-41dc-883c-b1a21e0f11f8-dshm\") pod \"premium-w0-a74d15-kserve-864f45fdfb-949gm\" (UID: \"eb0ced80-4c4f-41dc-883c-b1a21e0f11f8\") " pod="e2e-models-e2e-worker-w0-a74d15/premium-w0-a74d15-kserve-864f45fdfb-949gm" Oct 30 19:41:13.300031 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:13.300028 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb0ced80-4c4f-41dc-883c-b1a21e0f11f8-tmp-dir\") pod \"premium-w0-a74d15-kserve-864f45fdfb-949gm\" (UID: \"eb0ced80-4c4f-41dc-883c-b1a21e0f11f8\") " pod="e2e-models-e2e-worker-w0-a74d15/premium-w0-a74d15-kserve-864f45fdfb-949gm" Oct 30 19:41:13.300279 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:13.300150 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb0ced80-4c4f-41dc-883c-b1a21e0f11f8-tls-certs\") pod \"premium-w0-a74d15-kserve-864f45fdfb-949gm\" (UID: \"eb0ced80-4c4f-41dc-883c-b1a21e0f11f8\") " pod="e2e-models-e2e-worker-w0-a74d15/premium-w0-a74d15-kserve-864f45fdfb-949gm" Oct 30 19:41:13.300279 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:13.300231 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb0ced80-4c4f-41dc-883c-b1a21e0f11f8-model-cache\") pod \"premium-w0-a74d15-kserve-864f45fdfb-949gm\" (UID: \"eb0ced80-4c4f-41dc-883c-b1a21e0f11f8\") " pod="e2e-models-e2e-worker-w0-a74d15/premium-w0-a74d15-kserve-864f45fdfb-949gm" Oct 30 19:41:13.300279 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:13.300267 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv7f4\" (UniqueName: \"kubernetes.io/projected/eb0ced80-4c4f-41dc-883c-b1a21e0f11f8-kube-api-access-zv7f4\") pod \"premium-w0-a74d15-kserve-864f45fdfb-949gm\" (UID: \"eb0ced80-4c4f-41dc-883c-b1a21e0f11f8\") " pod="e2e-models-e2e-worker-w0-a74d15/premium-w0-a74d15-kserve-864f45fdfb-949gm" Oct 30 19:41:13.300463 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:13.300290 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb0ced80-4c4f-41dc-883c-b1a21e0f11f8-home\") pod \"premium-w0-a74d15-kserve-864f45fdfb-949gm\" (UID: \"eb0ced80-4c4f-41dc-883c-b1a21e0f11f8\") " pod="e2e-models-e2e-worker-w0-a74d15/premium-w0-a74d15-kserve-864f45fdfb-949gm" Oct 30 19:41:13.300546 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:13.300528 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb0ced80-4c4f-41dc-883c-b1a21e0f11f8-tmp-dir\") pod \"premium-w0-a74d15-kserve-864f45fdfb-949gm\" (UID: \"eb0ced80-4c4f-41dc-883c-b1a21e0f11f8\") " pod="e2e-models-e2e-worker-w0-a74d15/premium-w0-a74d15-kserve-864f45fdfb-949gm" Oct 30 19:41:13.300618 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:13.300599 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb0ced80-4c4f-41dc-883c-b1a21e0f11f8-model-cache\") pod \"premium-w0-a74d15-kserve-864f45fdfb-949gm\" (UID: \"eb0ced80-4c4f-41dc-883c-b1a21e0f11f8\") " pod="e2e-models-e2e-worker-w0-a74d15/premium-w0-a74d15-kserve-864f45fdfb-949gm" Oct 30 19:41:13.300749 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:13.300729 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb0ced80-4c4f-41dc-883c-b1a21e0f11f8-home\") pod \"premium-w0-a74d15-kserve-864f45fdfb-949gm\" (UID: \"eb0ced80-4c4f-41dc-883c-b1a21e0f11f8\") " pod="e2e-models-e2e-worker-w0-a74d15/premium-w0-a74d15-kserve-864f45fdfb-949gm" Oct 30 19:41:13.302442 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:13.302417 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb0ced80-4c4f-41dc-883c-b1a21e0f11f8-dshm\") pod \"premium-w0-a74d15-kserve-864f45fdfb-949gm\" (UID: \"eb0ced80-4c4f-41dc-883c-b1a21e0f11f8\") " pod="e2e-models-e2e-worker-w0-a74d15/premium-w0-a74d15-kserve-864f45fdfb-949gm" Oct 30 19:41:13.302705 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:13.302687 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb0ced80-4c4f-41dc-883c-b1a21e0f11f8-tls-certs\") pod \"premium-w0-a74d15-kserve-864f45fdfb-949gm\" (UID: \"eb0ced80-4c4f-41dc-883c-b1a21e0f11f8\") " pod="e2e-models-e2e-worker-w0-a74d15/premium-w0-a74d15-kserve-864f45fdfb-949gm" Oct 30 19:41:13.384833 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:13.384754 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv7f4\" (UniqueName: \"kubernetes.io/projected/eb0ced80-4c4f-41dc-883c-b1a21e0f11f8-kube-api-access-zv7f4\") pod \"premium-w0-a74d15-kserve-864f45fdfb-949gm\" (UID: \"eb0ced80-4c4f-41dc-883c-b1a21e0f11f8\") " pod="e2e-models-e2e-worker-w0-a74d15/premium-w0-a74d15-kserve-864f45fdfb-949gm" Oct 30 19:41:13.440279 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:13.440241 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a74d15/premium-w0-a74d15-kserve-864f45fdfb-949gm" Oct 30 19:41:13.843517 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:13.843491 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a74d15/premium-w0-a74d15-kserve-864f45fdfb-949gm"] Oct 30 19:41:13.845429 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:41:13.845404 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb0ced80_4c4f_41dc_883c_b1a21e0f11f8.slice/crio-405a23108c79c77532f530d636d8b7053ec1860e0aae282100f4027d06b8d217 WatchSource:0}: Error finding container 405a23108c79c77532f530d636d8b7053ec1860e0aae282100f4027d06b8d217: Status 404 returned error can't find the container with id 405a23108c79c77532f530d636d8b7053ec1860e0aae282100f4027d06b8d217 Oct 30 19:41:14.408046 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:14.408008 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a74d15/premium-w0-a74d15-kserve-864f45fdfb-949gm" event={"ID":"eb0ced80-4c4f-41dc-883c-b1a21e0f11f8","Type":"ContainerStarted","Data":"405a23108c79c77532f530d636d8b7053ec1860e0aae282100f4027d06b8d217"} Oct 30 19:41:16.187621 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:16.187577 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj"] Oct 30 19:41:16.192550 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:16.192521 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj" Oct 30 19:41:16.226514 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:16.226479 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8efc6802-ecd2-45b1-af43-799ca4e9bcb1-istio-data\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj\" (UID: \"8efc6802-ecd2-45b1-af43-799ca4e9bcb1\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj" Oct 30 19:41:16.226687 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:16.226533 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8efc6802-ecd2-45b1-af43-799ca4e9bcb1-credential-socket\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj\" (UID: \"8efc6802-ecd2-45b1-af43-799ca4e9bcb1\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj" Oct 30 19:41:16.226687 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:16.226619 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8efc6802-ecd2-45b1-af43-799ca4e9bcb1-workload-certs\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj\" (UID: \"8efc6802-ecd2-45b1-af43-799ca4e9bcb1\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj" Oct 30 19:41:16.226802 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:16.226684 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8efc6802-ecd2-45b1-af43-799ca4e9bcb1-istiod-ca-cert\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj\" (UID: \"8efc6802-ecd2-45b1-af43-799ca4e9bcb1\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj" Oct 30 19:41:16.226802 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:16.226782 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8efc6802-ecd2-45b1-af43-799ca4e9bcb1-istio-token\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj\" (UID: \"8efc6802-ecd2-45b1-af43-799ca4e9bcb1\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj" Oct 30 19:41:16.226879 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:16.226821 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfp56\" (UniqueName: \"kubernetes.io/projected/8efc6802-ecd2-45b1-af43-799ca4e9bcb1-kube-api-access-zfp56\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj\" (UID: \"8efc6802-ecd2-45b1-af43-799ca4e9bcb1\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj" Oct 30 19:41:16.226918 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:16.226893 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8efc6802-ecd2-45b1-af43-799ca4e9bcb1-workload-socket\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj\" (UID: \"8efc6802-ecd2-45b1-af43-799ca4e9bcb1\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj" Oct 30 19:41:16.226963 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:16.226934 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8efc6802-ecd2-45b1-af43-799ca4e9bcb1-istio-envoy\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj\" (UID: \"8efc6802-ecd2-45b1-af43-799ca4e9bcb1\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj" Oct 30 19:41:16.226963 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:16.226955 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8efc6802-ecd2-45b1-af43-799ca4e9bcb1-istio-podinfo\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj\" (UID: \"8efc6802-ecd2-45b1-af43-799ca4e9bcb1\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj" Oct 30 19:41:16.327804 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:16.327771 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8efc6802-ecd2-45b1-af43-799ca4e9bcb1-istio-podinfo\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj\" (UID: \"8efc6802-ecd2-45b1-af43-799ca4e9bcb1\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj" Oct 30 19:41:16.327963 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:16.327841 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8efc6802-ecd2-45b1-af43-799ca4e9bcb1-istio-data\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj\" (UID: \"8efc6802-ecd2-45b1-af43-799ca4e9bcb1\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj" Oct 30 19:41:16.327963 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:16.327862 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8efc6802-ecd2-45b1-af43-799ca4e9bcb1-credential-socket\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj\" (UID: \"8efc6802-ecd2-45b1-af43-799ca4e9bcb1\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj" Oct 30 19:41:16.327963 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:16.327881 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8efc6802-ecd2-45b1-af43-799ca4e9bcb1-workload-certs\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj\" (UID: \"8efc6802-ecd2-45b1-af43-799ca4e9bcb1\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj" Oct 30 19:41:16.327963 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:16.327943 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8efc6802-ecd2-45b1-af43-799ca4e9bcb1-istiod-ca-cert\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj\" (UID: \"8efc6802-ecd2-45b1-af43-799ca4e9bcb1\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj" Oct 30 19:41:16.328142 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:16.327970 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8efc6802-ecd2-45b1-af43-799ca4e9bcb1-istio-token\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj\" (UID: \"8efc6802-ecd2-45b1-af43-799ca4e9bcb1\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj" Oct 30 19:41:16.328142 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:16.327988 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfp56\" (UniqueName: \"kubernetes.io/projected/8efc6802-ecd2-45b1-af43-799ca4e9bcb1-kube-api-access-zfp56\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj\" (UID: \"8efc6802-ecd2-45b1-af43-799ca4e9bcb1\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj" Oct 30 19:41:16.328142 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:16.328019 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8efc6802-ecd2-45b1-af43-799ca4e9bcb1-workload-socket\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj\" (UID: \"8efc6802-ecd2-45b1-af43-799ca4e9bcb1\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj" Oct 30 19:41:16.328142 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:16.328047 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8efc6802-ecd2-45b1-af43-799ca4e9bcb1-istio-envoy\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj\" (UID: \"8efc6802-ecd2-45b1-af43-799ca4e9bcb1\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj" Oct 30 19:41:16.328409 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:16.328377 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8efc6802-ecd2-45b1-af43-799ca4e9bcb1-istio-data\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj\" (UID: \"8efc6802-ecd2-45b1-af43-799ca4e9bcb1\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj" Oct 30 19:41:16.328500 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:16.328416 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8efc6802-ecd2-45b1-af43-799ca4e9bcb1-credential-socket\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj\" (UID: \"8efc6802-ecd2-45b1-af43-799ca4e9bcb1\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj" Oct 30 19:41:16.328653 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:16.328624 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8efc6802-ecd2-45b1-af43-799ca4e9bcb1-workload-socket\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj\" (UID: \"8efc6802-ecd2-45b1-af43-799ca4e9bcb1\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj" Oct 30 19:41:16.328843 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:16.328811 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8efc6802-ecd2-45b1-af43-799ca4e9bcb1-istiod-ca-cert\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj\" (UID: \"8efc6802-ecd2-45b1-af43-799ca4e9bcb1\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj" Oct 30 19:41:16.328960 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:16.328813 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8efc6802-ecd2-45b1-af43-799ca4e9bcb1-workload-certs\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj\" (UID: \"8efc6802-ecd2-45b1-af43-799ca4e9bcb1\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj" Oct 30 19:41:16.330379 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:16.330323 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8efc6802-ecd2-45b1-af43-799ca4e9bcb1-istio-envoy\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj\" (UID: \"8efc6802-ecd2-45b1-af43-799ca4e9bcb1\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj" Oct 30 19:41:16.330803 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:16.330781 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8efc6802-ecd2-45b1-af43-799ca4e9bcb1-istio-podinfo\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj\" (UID: \"8efc6802-ecd2-45b1-af43-799ca4e9bcb1\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj" Oct 30 19:41:16.386463 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:16.386435 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8efc6802-ecd2-45b1-af43-799ca4e9bcb1-istio-token\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj\" (UID: \"8efc6802-ecd2-45b1-af43-799ca4e9bcb1\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj" Oct 30 19:41:16.386698 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:16.386679 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfp56\" (UniqueName: \"kubernetes.io/projected/8efc6802-ecd2-45b1-af43-799ca4e9bcb1-kube-api-access-zfp56\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj\" (UID: \"8efc6802-ecd2-45b1-af43-799ca4e9bcb1\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj" Oct 30 19:41:16.396335 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:16.396296 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj"] Oct 30 19:41:16.421966 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:16.421935 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a74d15/premium-w0-a74d15-kserve-864f45fdfb-949gm" event={"ID":"eb0ced80-4c4f-41dc-883c-b1a21e0f11f8","Type":"ContainerStarted","Data":"ab98b68c6892c684789cd6797cbeba1c02c6a1bab2ba608fd1b674c6e6351810"} Oct 30 19:41:16.441447 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:16.441375 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a74d15/premium-w0-a74d15-kserve-864f45fdfb-949gm" Oct 30 19:41:16.505041 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:16.505011 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj" Oct 30 19:41:16.534610 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:16.534560 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-a74d15/premium-w0-a74d15-kserve-864f45fdfb-949gm" podStartSLOduration=1.634510548 podStartE2EDuration="3.534545396s" totalImagesPullingTime="1.900034848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:16.533941772 +0000 UTC m=+980.052257249" watchObservedRunningTime="2026-10-30 19:41:16.534545396 +0000 UTC m=+980.052860852" Oct 30 19:41:16.925084 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:16.925059 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj"] Oct 30 19:41:16.927492 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:41:16.927459 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8efc6802_ecd2_45b1_af43_799ca4e9bcb1.slice/crio-4ab08da98feb787db61715e1d7a3cbcec7e3634f68550e51ea30064740a078dc WatchSource:0}: Error finding container 4ab08da98feb787db61715e1d7a3cbcec7e3634f68550e51ea30064740a078dc: Status 404 returned error can't find the container with id 4ab08da98feb787db61715e1d7a3cbcec7e3634f68550e51ea30064740a078dc Oct 30 19:41:16.929963 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:16.929925 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:41:16.930051 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:16.930018 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:41:16.930115 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:16.930076 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:41:17.428503 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:17.428459 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj" event={"ID":"8efc6802-ecd2-45b1-af43-799ca4e9bcb1","Type":"ContainerStarted","Data":"993f26fffefb36c7206d53e299629013acae48dfa108f4fe48988446cc2c349f"} Oct 30 19:41:17.428899 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:17.428531 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj" event={"ID":"8efc6802-ecd2-45b1-af43-799ca4e9bcb1","Type":"ContainerStarted","Data":"4ab08da98feb787db61715e1d7a3cbcec7e3634f68550e51ea30064740a078dc"} Oct 30 19:41:17.505864 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:17.505822 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj" Oct 30 19:41:17.507597 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:17.507569 4292 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Oct 30 19:41:17.507734 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:17.507620 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj" podUID="8efc6802-ecd2-45b1-af43-799ca4e9bcb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Oct 30 19:41:17.617535 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:17.617481 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj" podStartSLOduration=1.617466574 podStartE2EDuration="1.617466574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:17.588819995 +0000 UTC m=+981.107135483" watchObservedRunningTime="2026-10-30 19:41:17.617466574 +0000 UTC m=+981.135782032" Oct 30 19:41:18.505844 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:18.505810 4292 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Oct 30 19:41:18.506318 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:18.505870 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj" podUID="8efc6802-ecd2-45b1-af43-799ca4e9bcb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Oct 30 19:41:19.505741 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:19.505708 4292 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Oct 30 19:41:19.505927 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:19.505762 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj" podUID="8efc6802-ecd2-45b1-af43-799ca4e9bcb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Oct 30 19:41:20.505651 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:20.505618 4292 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Oct 30 19:41:20.505831 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:20.505671 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj" podUID="8efc6802-ecd2-45b1-af43-799ca4e9bcb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Oct 30 19:41:21.505746 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:21.505707 4292 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Oct 30 19:41:21.506152 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:21.505769 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj" podUID="8efc6802-ecd2-45b1-af43-799ca4e9bcb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Oct 30 19:41:22.506287 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:22.506251 4292 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Oct 30 19:41:22.506707 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:22.506304 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj" podUID="8efc6802-ecd2-45b1-af43-799ca4e9bcb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Oct 30 19:41:23.441305 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:23.441270 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-a74d15/premium-w0-a74d15-kserve-864f45fdfb-949gm" Oct 30 19:41:23.454701 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:23.454673 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-a74d15/premium-w0-a74d15-kserve-864f45fdfb-949gm" Oct 30 19:41:23.474245 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:23.474216 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a74d15/premium-w0-a74d15-kserve-864f45fdfb-949gm" Oct 30 19:41:23.510244 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:23.510215 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj" Oct 30 19:41:23.510663 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:23.510547 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj" Oct 30 19:41:23.511262 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:23.511240 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj" Oct 30 19:41:41.705043 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:41.705003 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-4ec609/premium-w4-4ec609-kserve-69f7c88bf7-rzmmt"] Oct 30 19:41:41.709757 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:41.709730 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ec609/premium-w4-4ec609-kserve-69f7c88bf7-rzmmt" Oct 30 19:41:41.771920 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:41.771884 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4ec609\"/\"premium-w4-4ec609-kserve-self-signed-certs\"" Oct 30 19:41:41.791388 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:41.791331 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb6be431-0a6c-4d81-b365-d7537f5d385a-model-cache\") pod \"premium-w4-4ec609-kserve-69f7c88bf7-rzmmt\" (UID: \"fb6be431-0a6c-4d81-b365-d7537f5d385a\") " pod="e2e-models-e2e-worker-w4-4ec609/premium-w4-4ec609-kserve-69f7c88bf7-rzmmt" Oct 30 19:41:41.791589 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:41.791448 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb6be431-0a6c-4d81-b365-d7537f5d385a-tls-certs\") pod \"premium-w4-4ec609-kserve-69f7c88bf7-rzmmt\" (UID: \"fb6be431-0a6c-4d81-b365-d7537f5d385a\") " pod="e2e-models-e2e-worker-w4-4ec609/premium-w4-4ec609-kserve-69f7c88bf7-rzmmt" Oct 30 19:41:41.791589 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:41.791512 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb6be431-0a6c-4d81-b365-d7537f5d385a-home\") pod \"premium-w4-4ec609-kserve-69f7c88bf7-rzmmt\" (UID: \"fb6be431-0a6c-4d81-b365-d7537f5d385a\") " pod="e2e-models-e2e-worker-w4-4ec609/premium-w4-4ec609-kserve-69f7c88bf7-rzmmt" Oct 30 19:41:41.791589 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:41.791542 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb6be431-0a6c-4d81-b365-d7537f5d385a-tmp-dir\") pod \"premium-w4-4ec609-kserve-69f7c88bf7-rzmmt\" (UID: \"fb6be431-0a6c-4d81-b365-d7537f5d385a\") " pod="e2e-models-e2e-worker-w4-4ec609/premium-w4-4ec609-kserve-69f7c88bf7-rzmmt" Oct 30 19:41:41.791714 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:41.791637 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhtnz\" (UniqueName: \"kubernetes.io/projected/fb6be431-0a6c-4d81-b365-d7537f5d385a-kube-api-access-lhtnz\") pod \"premium-w4-4ec609-kserve-69f7c88bf7-rzmmt\" (UID: \"fb6be431-0a6c-4d81-b365-d7537f5d385a\") " pod="e2e-models-e2e-worker-w4-4ec609/premium-w4-4ec609-kserve-69f7c88bf7-rzmmt" Oct 30 19:41:41.791714 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:41.791680 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb6be431-0a6c-4d81-b365-d7537f5d385a-dshm\") pod \"premium-w4-4ec609-kserve-69f7c88bf7-rzmmt\" (UID: \"fb6be431-0a6c-4d81-b365-d7537f5d385a\") " pod="e2e-models-e2e-worker-w4-4ec609/premium-w4-4ec609-kserve-69f7c88bf7-rzmmt" Oct 30 19:41:41.810812 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:41.810779 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4ec609\"/\"default-dockercfg-krcrx\"" Oct 30 19:41:41.810974 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:41.810923 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-4ec609\"/\"kube-root-ca.crt\"" Oct 30 19:41:41.812787 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:41.812763 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4ec609/premium-w4-4ec609-kserve-69f7c88bf7-rzmmt"] Oct 30 19:41:41.861148 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:41.861110 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-4ec609\"/\"openshift-service-ca.crt\"" Oct 30 19:41:41.892478 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:41.892430 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhtnz\" (UniqueName: \"kubernetes.io/projected/fb6be431-0a6c-4d81-b365-d7537f5d385a-kube-api-access-lhtnz\") pod \"premium-w4-4ec609-kserve-69f7c88bf7-rzmmt\" (UID: \"fb6be431-0a6c-4d81-b365-d7537f5d385a\") " pod="e2e-models-e2e-worker-w4-4ec609/premium-w4-4ec609-kserve-69f7c88bf7-rzmmt" Oct 30 19:41:41.892693 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:41.892518 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb6be431-0a6c-4d81-b365-d7537f5d385a-dshm\") pod \"premium-w4-4ec609-kserve-69f7c88bf7-rzmmt\" (UID: \"fb6be431-0a6c-4d81-b365-d7537f5d385a\") " pod="e2e-models-e2e-worker-w4-4ec609/premium-w4-4ec609-kserve-69f7c88bf7-rzmmt" Oct 30 19:41:41.892693 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:41.892575 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb6be431-0a6c-4d81-b365-d7537f5d385a-model-cache\") pod \"premium-w4-4ec609-kserve-69f7c88bf7-rzmmt\" (UID: \"fb6be431-0a6c-4d81-b365-d7537f5d385a\") " pod="e2e-models-e2e-worker-w4-4ec609/premium-w4-4ec609-kserve-69f7c88bf7-rzmmt" Oct 30 19:41:41.892693 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:41.892610 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb6be431-0a6c-4d81-b365-d7537f5d385a-tls-certs\") pod \"premium-w4-4ec609-kserve-69f7c88bf7-rzmmt\" (UID: \"fb6be431-0a6c-4d81-b365-d7537f5d385a\") " pod="e2e-models-e2e-worker-w4-4ec609/premium-w4-4ec609-kserve-69f7c88bf7-rzmmt" Oct 30 19:41:41.892693 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:41.892651 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb6be431-0a6c-4d81-b365-d7537f5d385a-home\") pod \"premium-w4-4ec609-kserve-69f7c88bf7-rzmmt\" (UID: \"fb6be431-0a6c-4d81-b365-d7537f5d385a\") " pod="e2e-models-e2e-worker-w4-4ec609/premium-w4-4ec609-kserve-69f7c88bf7-rzmmt" Oct 30 19:41:41.892693 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:41.892677 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb6be431-0a6c-4d81-b365-d7537f5d385a-tmp-dir\") pod \"premium-w4-4ec609-kserve-69f7c88bf7-rzmmt\" (UID: \"fb6be431-0a6c-4d81-b365-d7537f5d385a\") " pod="e2e-models-e2e-worker-w4-4ec609/premium-w4-4ec609-kserve-69f7c88bf7-rzmmt" Oct 30 19:41:41.893086 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:41.893049 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb6be431-0a6c-4d81-b365-d7537f5d385a-model-cache\") pod \"premium-w4-4ec609-kserve-69f7c88bf7-rzmmt\" (UID: \"fb6be431-0a6c-4d81-b365-d7537f5d385a\") " pod="e2e-models-e2e-worker-w4-4ec609/premium-w4-4ec609-kserve-69f7c88bf7-rzmmt" Oct 30 19:41:41.893337 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:41.893312 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb6be431-0a6c-4d81-b365-d7537f5d385a-tmp-dir\") pod \"premium-w4-4ec609-kserve-69f7c88bf7-rzmmt\" (UID: \"fb6be431-0a6c-4d81-b365-d7537f5d385a\") " pod="e2e-models-e2e-worker-w4-4ec609/premium-w4-4ec609-kserve-69f7c88bf7-rzmmt" Oct 30 19:41:41.893457 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:41.893383 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb6be431-0a6c-4d81-b365-d7537f5d385a-home\") pod \"premium-w4-4ec609-kserve-69f7c88bf7-rzmmt\" (UID: \"fb6be431-0a6c-4d81-b365-d7537f5d385a\") " pod="e2e-models-e2e-worker-w4-4ec609/premium-w4-4ec609-kserve-69f7c88bf7-rzmmt" Oct 30 19:41:41.894948 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:41.894931 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb6be431-0a6c-4d81-b365-d7537f5d385a-dshm\") pod \"premium-w4-4ec609-kserve-69f7c88bf7-rzmmt\" (UID: \"fb6be431-0a6c-4d81-b365-d7537f5d385a\") " pod="e2e-models-e2e-worker-w4-4ec609/premium-w4-4ec609-kserve-69f7c88bf7-rzmmt" Oct 30 19:41:41.895637 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:41.895614 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb6be431-0a6c-4d81-b365-d7537f5d385a-tls-certs\") pod \"premium-w4-4ec609-kserve-69f7c88bf7-rzmmt\" (UID: \"fb6be431-0a6c-4d81-b365-d7537f5d385a\") " pod="e2e-models-e2e-worker-w4-4ec609/premium-w4-4ec609-kserve-69f7c88bf7-rzmmt" Oct 30 19:41:41.953981 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:41.953947 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhtnz\" (UniqueName: \"kubernetes.io/projected/fb6be431-0a6c-4d81-b365-d7537f5d385a-kube-api-access-lhtnz\") pod \"premium-w4-4ec609-kserve-69f7c88bf7-rzmmt\" (UID: \"fb6be431-0a6c-4d81-b365-d7537f5d385a\") " pod="e2e-models-e2e-worker-w4-4ec609/premium-w4-4ec609-kserve-69f7c88bf7-rzmmt" Oct 30 19:41:42.022124 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:42.022091 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ec609/premium-w4-4ec609-kserve-69f7c88bf7-rzmmt" Oct 30 19:41:42.447916 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:42.447890 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4ec609/premium-w4-4ec609-kserve-69f7c88bf7-rzmmt"] Oct 30 19:41:42.450731 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:41:42.450701 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb6be431_0a6c_4d81_b365_d7537f5d385a.slice/crio-81c9b83d4a5be10cf3c74d94d1f56b275063c5611034afde5af78ca472d7704d WatchSource:0}: Error finding container 81c9b83d4a5be10cf3c74d94d1f56b275063c5611034afde5af78ca472d7704d: Status 404 returned error can't find the container with id 81c9b83d4a5be10cf3c74d94d1f56b275063c5611034afde5af78ca472d7704d Oct 30 19:41:42.550028 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:42.549985 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ec609/premium-w4-4ec609-kserve-69f7c88bf7-rzmmt" event={"ID":"fb6be431-0a6c-4d81-b365-d7537f5d385a","Type":"ContainerStarted","Data":"85a25601e116c5f2a5baa471fe6a6af436c3b2c8ea9136b98009fe3e9db74cf6"} Oct 30 19:41:42.550028 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:42.550034 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ec609/premium-w4-4ec609-kserve-69f7c88bf7-rzmmt" event={"ID":"fb6be431-0a6c-4d81-b365-d7537f5d385a","Type":"ContainerStarted","Data":"81c9b83d4a5be10cf3c74d94d1f56b275063c5611034afde5af78ca472d7704d"} Oct 30 19:41:42.688968 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:42.688912 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-4ec609/premium-w4-4ec609-kserve-69f7c88bf7-rzmmt" podStartSLOduration=1.688894828 podStartE2EDuration="1.688894828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:42.687754823 +0000 UTC m=+1006.206070291" watchObservedRunningTime="2026-10-30 19:41:42.688894828 +0000 UTC m=+1006.207210285" Oct 30 19:41:43.022439 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:43.022400 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ec609/premium-w4-4ec609-kserve-69f7c88bf7-rzmmt" Oct 30 19:41:43.222126 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:41:43.222086 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:41:50.362051 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:50.362016 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-98a19b/simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r"] Oct 30 19:41:50.365474 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:50.365455 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-98a19b/simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r" Oct 30 19:41:50.467865 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:50.467835 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-98a19b\"/\"openshift-service-ca.crt\"" Oct 30 19:41:50.468078 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:50.467890 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-98a19b\"/\"simulator-w1-98a19b-kserve-self-signed-certs\"" Oct 30 19:41:50.468078 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:50.468007 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-98a19b\"/\"default-dockercfg-hntn4\"" Oct 30 19:41:50.468078 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:50.468024 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-98a19b\"/\"kube-root-ca.crt\"" Oct 30 19:41:50.470035 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:50.470015 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-98a19b/simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r"] Oct 30 19:41:50.475046 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:50.475023 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f35e32e-1f09-413d-a364-14b8ac1019f2-home\") pod \"simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r\" (UID: \"1f35e32e-1f09-413d-a364-14b8ac1019f2\") " pod="e2e-models-e2e-worker-w1-98a19b/simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r" Oct 30 19:41:50.475186 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:50.475068 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f35e32e-1f09-413d-a364-14b8ac1019f2-tmp-dir\") pod \"simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r\" (UID: \"1f35e32e-1f09-413d-a364-14b8ac1019f2\") " pod="e2e-models-e2e-worker-w1-98a19b/simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r" Oct 30 19:41:50.475186 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:50.475137 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f35e32e-1f09-413d-a364-14b8ac1019f2-model-cache\") pod \"simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r\" (UID: \"1f35e32e-1f09-413d-a364-14b8ac1019f2\") " pod="e2e-models-e2e-worker-w1-98a19b/simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r" Oct 30 19:41:50.475186 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:50.475170 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f35e32e-1f09-413d-a364-14b8ac1019f2-tls-certs\") pod \"simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r\" (UID: \"1f35e32e-1f09-413d-a364-14b8ac1019f2\") " pod="e2e-models-e2e-worker-w1-98a19b/simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r" Oct 30 19:41:50.475335 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:50.475199 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdsph\" (UniqueName: \"kubernetes.io/projected/1f35e32e-1f09-413d-a364-14b8ac1019f2-kube-api-access-jdsph\") pod \"simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r\" (UID: \"1f35e32e-1f09-413d-a364-14b8ac1019f2\") " pod="e2e-models-e2e-worker-w1-98a19b/simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r" Oct 30 19:41:50.475335 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:50.475322 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f35e32e-1f09-413d-a364-14b8ac1019f2-dshm\") pod \"simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r\" (UID: \"1f35e32e-1f09-413d-a364-14b8ac1019f2\") " pod="e2e-models-e2e-worker-w1-98a19b/simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r" Oct 30 19:41:50.576259 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:50.576220 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f35e32e-1f09-413d-a364-14b8ac1019f2-model-cache\") pod \"simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r\" (UID: \"1f35e32e-1f09-413d-a364-14b8ac1019f2\") " pod="e2e-models-e2e-worker-w1-98a19b/simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r" Oct 30 19:41:50.576444 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:50.576268 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f35e32e-1f09-413d-a364-14b8ac1019f2-tls-certs\") pod \"simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r\" (UID: \"1f35e32e-1f09-413d-a364-14b8ac1019f2\") " pod="e2e-models-e2e-worker-w1-98a19b/simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r" Oct 30 19:41:50.576444 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:50.576297 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdsph\" (UniqueName: \"kubernetes.io/projected/1f35e32e-1f09-413d-a364-14b8ac1019f2-kube-api-access-jdsph\") pod \"simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r\" (UID: \"1f35e32e-1f09-413d-a364-14b8ac1019f2\") " pod="e2e-models-e2e-worker-w1-98a19b/simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r" Oct 30 19:41:50.576444 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:50.576390 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f35e32e-1f09-413d-a364-14b8ac1019f2-dshm\") pod \"simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r\" (UID: \"1f35e32e-1f09-413d-a364-14b8ac1019f2\") " pod="e2e-models-e2e-worker-w1-98a19b/simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r" Oct 30 19:41:50.576641 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:50.576486 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f35e32e-1f09-413d-a364-14b8ac1019f2-home\") pod \"simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r\" (UID: \"1f35e32e-1f09-413d-a364-14b8ac1019f2\") " pod="e2e-models-e2e-worker-w1-98a19b/simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r" Oct 30 19:41:50.576641 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:50.576549 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f35e32e-1f09-413d-a364-14b8ac1019f2-tmp-dir\") pod \"simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r\" (UID: \"1f35e32e-1f09-413d-a364-14b8ac1019f2\") " pod="e2e-models-e2e-worker-w1-98a19b/simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r" Oct 30 19:41:50.576749 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:50.576688 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f35e32e-1f09-413d-a364-14b8ac1019f2-model-cache\") pod \"simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r\" (UID: \"1f35e32e-1f09-413d-a364-14b8ac1019f2\") " pod="e2e-models-e2e-worker-w1-98a19b/simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r" Oct 30 19:41:50.576898 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:50.576876 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f35e32e-1f09-413d-a364-14b8ac1019f2-tmp-dir\") pod \"simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r\" (UID: \"1f35e32e-1f09-413d-a364-14b8ac1019f2\") " pod="e2e-models-e2e-worker-w1-98a19b/simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r" Oct 30 19:41:50.576976 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:50.576955 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f35e32e-1f09-413d-a364-14b8ac1019f2-home\") pod \"simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r\" (UID: \"1f35e32e-1f09-413d-a364-14b8ac1019f2\") " pod="e2e-models-e2e-worker-w1-98a19b/simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r" Oct 30 19:41:50.578966 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:50.578936 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f35e32e-1f09-413d-a364-14b8ac1019f2-dshm\") pod \"simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r\" (UID: \"1f35e32e-1f09-413d-a364-14b8ac1019f2\") " pod="e2e-models-e2e-worker-w1-98a19b/simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r" Oct 30 19:41:50.579406 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:50.579382 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f35e32e-1f09-413d-a364-14b8ac1019f2-tls-certs\") pod \"simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r\" (UID: \"1f35e32e-1f09-413d-a364-14b8ac1019f2\") " pod="e2e-models-e2e-worker-w1-98a19b/simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r" Oct 30 19:41:50.643940 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:50.643858 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdsph\" (UniqueName: \"kubernetes.io/projected/1f35e32e-1f09-413d-a364-14b8ac1019f2-kube-api-access-jdsph\") pod \"simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r\" (UID: \"1f35e32e-1f09-413d-a364-14b8ac1019f2\") " pod="e2e-models-e2e-worker-w1-98a19b/simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r" Oct 30 19:41:50.676677 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:50.676636 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-98a19b/simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r" Oct 30 19:41:51.051072 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:51.051040 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-98a19b/simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r"] Oct 30 19:41:51.054152 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:41:51.054121 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f35e32e_1f09_413d_a364_14b8ac1019f2.slice/crio-c441d1a5aba7925a7addd7bc28b36005309a58d63d73cddcb8dcb033b3973b9a WatchSource:0}: Error finding container c441d1a5aba7925a7addd7bc28b36005309a58d63d73cddcb8dcb033b3973b9a: Status 404 returned error can't find the container with id c441d1a5aba7925a7addd7bc28b36005309a58d63d73cddcb8dcb033b3973b9a Oct 30 19:41:51.593652 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:51.593615 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-98a19b/simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r" event={"ID":"1f35e32e-1f09-413d-a364-14b8ac1019f2","Type":"ContainerStarted","Data":"7d0089e0b6f5fde3277d4efb2b31ac19a0e1f76bdc2332548fe9b4293ddbb15b"} Oct 30 19:41:51.594236 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:51.593667 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-98a19b/simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r" event={"ID":"1f35e32e-1f09-413d-a364-14b8ac1019f2","Type":"ContainerStarted","Data":"c441d1a5aba7925a7addd7bc28b36005309a58d63d73cddcb8dcb033b3973b9a"} Oct 30 19:41:51.677154 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:51.677123 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-98a19b/simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r" Oct 30 19:41:51.704964 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:51.704914 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-98a19b/simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r" podStartSLOduration=1.704900496 podStartE2EDuration="1.704900496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:51.70444062 +0000 UTC m=+1015.222756103" watchObservedRunningTime="2026-10-30 19:41:51.704900496 +0000 UTC m=+1015.223215951" Oct 30 19:41:52.022447 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:52.022412 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-4ec609/premium-w4-4ec609-kserve-69f7c88bf7-rzmmt" Oct 30 19:41:52.035225 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:52.035199 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-4ec609/premium-w4-4ec609-kserve-69f7c88bf7-rzmmt" Oct 30 19:41:52.609371 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:41:52.609324 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ec609/premium-w4-4ec609-kserve-69f7c88bf7-rzmmt" Oct 30 19:42:00.676920 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:42:00.676888 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-98a19b/simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r" Oct 30 19:42:00.698275 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:42:00.698244 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-98a19b/simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r" Oct 30 19:42:01.676816 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:42:01.676786 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-98a19b/simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r" Oct 30 19:42:13.220553 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:42:13.220469 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:42:21.256817 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:42:21.256784 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-bbfa07/premium-w3-bbfa07-kserve-5675444c9-x42d7"] Oct 30 19:42:21.260559 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:42:21.260526 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bbfa07/premium-w3-bbfa07-kserve-5675444c9-x42d7" Oct 30 19:42:21.357203 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:42:21.357170 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-bbfa07\"/\"premium-w3-bbfa07-kserve-self-signed-certs\"" Oct 30 19:42:21.357203 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:42:21.357175 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-bbfa07\"/\"default-dockercfg-r6p6l\"" Oct 30 19:42:21.357773 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:42:21.357757 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-bbfa07\"/\"kube-root-ca.crt\"" Oct 30 19:42:21.357813 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:42:21.357797 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-bbfa07\"/\"openshift-service-ca.crt\"" Oct 30 19:42:21.359278 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:42:21.359255 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-bbfa07/premium-w3-bbfa07-kserve-5675444c9-x42d7"] Oct 30 19:42:21.389730 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:42:21.389699 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0b75746-c06c-407d-8f61-c795c519aad6-home\") pod \"premium-w3-bbfa07-kserve-5675444c9-x42d7\" (UID: \"e0b75746-c06c-407d-8f61-c795c519aad6\") " pod="e2e-models-e2e-worker-w3-bbfa07/premium-w3-bbfa07-kserve-5675444c9-x42d7" Oct 30 19:42:21.389730 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:42:21.389732 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0b75746-c06c-407d-8f61-c795c519aad6-tls-certs\") pod \"premium-w3-bbfa07-kserve-5675444c9-x42d7\" (UID: \"e0b75746-c06c-407d-8f61-c795c519aad6\") " pod="e2e-models-e2e-worker-w3-bbfa07/premium-w3-bbfa07-kserve-5675444c9-x42d7" Oct 30 19:42:21.389912 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:42:21.389829 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0b75746-c06c-407d-8f61-c795c519aad6-model-cache\") pod \"premium-w3-bbfa07-kserve-5675444c9-x42d7\" (UID: \"e0b75746-c06c-407d-8f61-c795c519aad6\") " pod="e2e-models-e2e-worker-w3-bbfa07/premium-w3-bbfa07-kserve-5675444c9-x42d7" Oct 30 19:42:21.389912 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:42:21.389881 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knc5n\" (UniqueName: \"kubernetes.io/projected/e0b75746-c06c-407d-8f61-c795c519aad6-kube-api-access-knc5n\") pod \"premium-w3-bbfa07-kserve-5675444c9-x42d7\" (UID: \"e0b75746-c06c-407d-8f61-c795c519aad6\") " pod="e2e-models-e2e-worker-w3-bbfa07/premium-w3-bbfa07-kserve-5675444c9-x42d7" Oct 30 19:42:21.389982 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:42:21.389923 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0b75746-c06c-407d-8f61-c795c519aad6-tmp-dir\") pod \"premium-w3-bbfa07-kserve-5675444c9-x42d7\" (UID: \"e0b75746-c06c-407d-8f61-c795c519aad6\") " pod="e2e-models-e2e-worker-w3-bbfa07/premium-w3-bbfa07-kserve-5675444c9-x42d7" Oct 30 19:42:21.390023 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:42:21.390006 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0b75746-c06c-407d-8f61-c795c519aad6-dshm\") pod \"premium-w3-bbfa07-kserve-5675444c9-x42d7\" (UID: \"e0b75746-c06c-407d-8f61-c795c519aad6\") " pod="e2e-models-e2e-worker-w3-bbfa07/premium-w3-bbfa07-kserve-5675444c9-x42d7" Oct 30 19:42:21.490786 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:42:21.490745 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0b75746-c06c-407d-8f61-c795c519aad6-tmp-dir\") pod \"premium-w3-bbfa07-kserve-5675444c9-x42d7\" (UID: \"e0b75746-c06c-407d-8f61-c795c519aad6\") " pod="e2e-models-e2e-worker-w3-bbfa07/premium-w3-bbfa07-kserve-5675444c9-x42d7" Oct 30 19:42:21.490952 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:42:21.490828 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0b75746-c06c-407d-8f61-c795c519aad6-dshm\") pod \"premium-w3-bbfa07-kserve-5675444c9-x42d7\" (UID: \"e0b75746-c06c-407d-8f61-c795c519aad6\") " pod="e2e-models-e2e-worker-w3-bbfa07/premium-w3-bbfa07-kserve-5675444c9-x42d7" Oct 30 19:42:21.490952 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:42:21.490893 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0b75746-c06c-407d-8f61-c795c519aad6-home\") pod \"premium-w3-bbfa07-kserve-5675444c9-x42d7\" (UID: \"e0b75746-c06c-407d-8f61-c795c519aad6\") " pod="e2e-models-e2e-worker-w3-bbfa07/premium-w3-bbfa07-kserve-5675444c9-x42d7" Oct 30 19:42:21.490952 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:42:21.490922 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0b75746-c06c-407d-8f61-c795c519aad6-tls-certs\") pod \"premium-w3-bbfa07-kserve-5675444c9-x42d7\" (UID: \"e0b75746-c06c-407d-8f61-c795c519aad6\") " pod="e2e-models-e2e-worker-w3-bbfa07/premium-w3-bbfa07-kserve-5675444c9-x42d7" Oct 30 19:42:21.491100 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:42:21.490991 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0b75746-c06c-407d-8f61-c795c519aad6-model-cache\") pod \"premium-w3-bbfa07-kserve-5675444c9-x42d7\" (UID: \"e0b75746-c06c-407d-8f61-c795c519aad6\") " pod="e2e-models-e2e-worker-w3-bbfa07/premium-w3-bbfa07-kserve-5675444c9-x42d7" Oct 30 19:42:21.491100 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:42:21.491032 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knc5n\" (UniqueName: \"kubernetes.io/projected/e0b75746-c06c-407d-8f61-c795c519aad6-kube-api-access-knc5n\") pod \"premium-w3-bbfa07-kserve-5675444c9-x42d7\" (UID: \"e0b75746-c06c-407d-8f61-c795c519aad6\") " pod="e2e-models-e2e-worker-w3-bbfa07/premium-w3-bbfa07-kserve-5675444c9-x42d7" Oct 30 19:42:21.491206 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:42:21.491149 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0b75746-c06c-407d-8f61-c795c519aad6-tmp-dir\") pod \"premium-w3-bbfa07-kserve-5675444c9-x42d7\" (UID: \"e0b75746-c06c-407d-8f61-c795c519aad6\") " pod="e2e-models-e2e-worker-w3-bbfa07/premium-w3-bbfa07-kserve-5675444c9-x42d7" Oct 30 19:42:21.491315 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:42:21.491274 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0b75746-c06c-407d-8f61-c795c519aad6-home\") pod \"premium-w3-bbfa07-kserve-5675444c9-x42d7\" (UID: \"e0b75746-c06c-407d-8f61-c795c519aad6\") " pod="e2e-models-e2e-worker-w3-bbfa07/premium-w3-bbfa07-kserve-5675444c9-x42d7" Oct 30 19:42:21.491499 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:42:21.491456 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0b75746-c06c-407d-8f61-c795c519aad6-model-cache\") pod \"premium-w3-bbfa07-kserve-5675444c9-x42d7\" (UID: \"e0b75746-c06c-407d-8f61-c795c519aad6\") " pod="e2e-models-e2e-worker-w3-bbfa07/premium-w3-bbfa07-kserve-5675444c9-x42d7" Oct 30 19:42:21.493213 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:42:21.493183 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0b75746-c06c-407d-8f61-c795c519aad6-dshm\") pod \"premium-w3-bbfa07-kserve-5675444c9-x42d7\" (UID: \"e0b75746-c06c-407d-8f61-c795c519aad6\") " pod="e2e-models-e2e-worker-w3-bbfa07/premium-w3-bbfa07-kserve-5675444c9-x42d7" Oct 30 19:42:21.493447 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:42:21.493430 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0b75746-c06c-407d-8f61-c795c519aad6-tls-certs\") pod \"premium-w3-bbfa07-kserve-5675444c9-x42d7\" (UID: \"e0b75746-c06c-407d-8f61-c795c519aad6\") " pod="e2e-models-e2e-worker-w3-bbfa07/premium-w3-bbfa07-kserve-5675444c9-x42d7" Oct 30 19:42:21.550131 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:42:21.550093 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knc5n\" (UniqueName: \"kubernetes.io/projected/e0b75746-c06c-407d-8f61-c795c519aad6-kube-api-access-knc5n\") pod \"premium-w3-bbfa07-kserve-5675444c9-x42d7\" (UID: \"e0b75746-c06c-407d-8f61-c795c519aad6\") " pod="e2e-models-e2e-worker-w3-bbfa07/premium-w3-bbfa07-kserve-5675444c9-x42d7" Oct 30 19:42:21.572816 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:42:21.572776 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bbfa07/premium-w3-bbfa07-kserve-5675444c9-x42d7" Oct 30 19:42:21.981148 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:42:21.981124 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-bbfa07/premium-w3-bbfa07-kserve-5675444c9-x42d7"] Oct 30 19:42:21.983788 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:42:21.983760 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode0b75746_c06c_407d_8f61_c795c519aad6.slice/crio-5a666776a564544573674eb2e64b06143d021eb10006622e7378cb3b00dc3c14 WatchSource:0}: Error finding container 5a666776a564544573674eb2e64b06143d021eb10006622e7378cb3b00dc3c14: Status 404 returned error can't find the container with id 5a666776a564544573674eb2e64b06143d021eb10006622e7378cb3b00dc3c14 Oct 30 19:42:22.755619 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:42:22.755577 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bbfa07/premium-w3-bbfa07-kserve-5675444c9-x42d7" event={"ID":"e0b75746-c06c-407d-8f61-c795c519aad6","Type":"ContainerStarted","Data":"3d73da7a92814ad72ece1fe671bfec46d0335b3e7f2c12d766e01be7ec48c87e"} Oct 30 19:42:22.756035 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:42:22.755631 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bbfa07/premium-w3-bbfa07-kserve-5675444c9-x42d7" event={"ID":"e0b75746-c06c-407d-8f61-c795c519aad6","Type":"ContainerStarted","Data":"5a666776a564544573674eb2e64b06143d021eb10006622e7378cb3b00dc3c14"} Oct 30 19:42:22.902557 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:42:22.902500 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-bbfa07/premium-w3-bbfa07-kserve-5675444c9-x42d7" podStartSLOduration=1.9024868769999999 podStartE2EDuration="1.902486877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:22.901611154 +0000 UTC m=+1046.419926630" watchObservedRunningTime="2026-10-30 19:42:22.902486877 +0000 UTC m=+1046.420802333" Oct 30 19:42:23.572967 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:42:23.572938 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bbfa07/premium-w3-bbfa07-kserve-5675444c9-x42d7" Oct 30 19:42:31.573712 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:42:31.573671 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-bbfa07/premium-w3-bbfa07-kserve-5675444c9-x42d7" Oct 30 19:42:31.586379 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:42:31.586336 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-bbfa07/premium-w3-bbfa07-kserve-5675444c9-x42d7" Oct 30 19:42:31.807151 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:42:31.807118 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bbfa07/premium-w3-bbfa07-kserve-5675444c9-x42d7" Oct 30 19:42:43.223165 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:42:43.223121 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:43:13.219475 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:43:13.219436 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:43:37.074059 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:37.074020 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-788b6c6b4d-59ggs"] Oct 30 19:43:37.077767 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:37.077746 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-788b6c6b4d-59ggs" Oct 30 19:43:37.173858 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:37.173823 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-449cc5558c4a\"" Oct 30 19:43:37.173858 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:37.173859 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 30 19:43:37.174115 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:37.173822 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 30 19:43:37.174115 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:37.173980 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 30 19:43:37.174115 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:37.174011 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 30 19:43:37.174115 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:37.174032 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-qvs4q\"" Oct 30 19:43:37.174115 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:37.174067 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-294332af605e\"" Oct 30 19:43:37.176431 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:37.176414 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-788b6c6b4d-59ggs"] Oct 30 19:43:37.265855 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:37.265824 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a179efae-1f42-4b73-b2d7-70f1474c3c9f-plugins-config-volume\") pod \"payload-processing-external-model-788b6c6b4d-59ggs\" (UID: \"a179efae-1f42-4b73-b2d7-70f1474c3c9f\") " pod="models-as-a-service/payload-processing-external-model-788b6c6b4d-59ggs" Oct 30 19:43:37.266021 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:37.265860 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2dqr\" (UniqueName: \"kubernetes.io/projected/a179efae-1f42-4b73-b2d7-70f1474c3c9f-kube-api-access-t2dqr\") pod \"payload-processing-external-model-788b6c6b4d-59ggs\" (UID: \"a179efae-1f42-4b73-b2d7-70f1474c3c9f\") " pod="models-as-a-service/payload-processing-external-model-788b6c6b4d-59ggs" Oct 30 19:43:37.266021 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:37.265901 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a179efae-1f42-4b73-b2d7-70f1474c3c9f-routing-overlay-volume\") pod \"payload-processing-external-model-788b6c6b4d-59ggs\" (UID: \"a179efae-1f42-4b73-b2d7-70f1474c3c9f\") " pod="models-as-a-service/payload-processing-external-model-788b6c6b4d-59ggs" Oct 30 19:43:37.266021 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:37.266006 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a179efae-1f42-4b73-b2d7-70f1474c3c9f-provider-credentials\") pod \"payload-processing-external-model-788b6c6b4d-59ggs\" (UID: \"a179efae-1f42-4b73-b2d7-70f1474c3c9f\") " pod="models-as-a-service/payload-processing-external-model-788b6c6b4d-59ggs" Oct 30 19:43:37.367599 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:37.367513 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a179efae-1f42-4b73-b2d7-70f1474c3c9f-routing-overlay-volume\") pod \"payload-processing-external-model-788b6c6b4d-59ggs\" (UID: \"a179efae-1f42-4b73-b2d7-70f1474c3c9f\") " pod="models-as-a-service/payload-processing-external-model-788b6c6b4d-59ggs" Oct 30 19:43:37.367599 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:37.367569 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a179efae-1f42-4b73-b2d7-70f1474c3c9f-provider-credentials\") pod \"payload-processing-external-model-788b6c6b4d-59ggs\" (UID: \"a179efae-1f42-4b73-b2d7-70f1474c3c9f\") " pod="models-as-a-service/payload-processing-external-model-788b6c6b4d-59ggs" Oct 30 19:43:37.367790 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:37.367682 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a179efae-1f42-4b73-b2d7-70f1474c3c9f-plugins-config-volume\") pod \"payload-processing-external-model-788b6c6b4d-59ggs\" (UID: \"a179efae-1f42-4b73-b2d7-70f1474c3c9f\") " pod="models-as-a-service/payload-processing-external-model-788b6c6b4d-59ggs" Oct 30 19:43:37.367790 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:37.367714 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2dqr\" (UniqueName: \"kubernetes.io/projected/a179efae-1f42-4b73-b2d7-70f1474c3c9f-kube-api-access-t2dqr\") pod \"payload-processing-external-model-788b6c6b4d-59ggs\" (UID: \"a179efae-1f42-4b73-b2d7-70f1474c3c9f\") " pod="models-as-a-service/payload-processing-external-model-788b6c6b4d-59ggs" Oct 30 19:43:37.368310 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:37.368285 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a179efae-1f42-4b73-b2d7-70f1474c3c9f-routing-overlay-volume\") pod \"payload-processing-external-model-788b6c6b4d-59ggs\" (UID: \"a179efae-1f42-4b73-b2d7-70f1474c3c9f\") " pod="models-as-a-service/payload-processing-external-model-788b6c6b4d-59ggs" Oct 30 19:43:37.368440 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:37.368421 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a179efae-1f42-4b73-b2d7-70f1474c3c9f-plugins-config-volume\") pod \"payload-processing-external-model-788b6c6b4d-59ggs\" (UID: \"a179efae-1f42-4b73-b2d7-70f1474c3c9f\") " pod="models-as-a-service/payload-processing-external-model-788b6c6b4d-59ggs" Oct 30 19:43:37.370262 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:37.370237 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a179efae-1f42-4b73-b2d7-70f1474c3c9f-provider-credentials\") pod \"payload-processing-external-model-788b6c6b4d-59ggs\" (UID: \"a179efae-1f42-4b73-b2d7-70f1474c3c9f\") " pod="models-as-a-service/payload-processing-external-model-788b6c6b4d-59ggs" Oct 30 19:43:37.426315 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:37.426279 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2dqr\" (UniqueName: \"kubernetes.io/projected/a179efae-1f42-4b73-b2d7-70f1474c3c9f-kube-api-access-t2dqr\") pod \"payload-processing-external-model-788b6c6b4d-59ggs\" (UID: \"a179efae-1f42-4b73-b2d7-70f1474c3c9f\") " pod="models-as-a-service/payload-processing-external-model-788b6c6b4d-59ggs" Oct 30 19:43:37.691407 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:37.691296 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-788b6c6b4d-59ggs" Oct 30 19:43:38.056548 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:38.056519 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-788b6c6b4d-59ggs"] Oct 30 19:43:38.058983 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:43:38.058949 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda179efae_1f42_4b73_b2d7_70f1474c3c9f.slice/crio-7623105fb7e3da7709ac11895d4789931ba8a97eadfe5e42f7fe5dbb400b225e WatchSource:0}: Error finding container 7623105fb7e3da7709ac11895d4789931ba8a97eadfe5e42f7fe5dbb400b225e: Status 404 returned error can't find the container with id 7623105fb7e3da7709ac11895d4789931ba8a97eadfe5e42f7fe5dbb400b225e Oct 30 19:43:38.125919 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:38.125884 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-788b6c6b4d-59ggs" event={"ID":"a179efae-1f42-4b73-b2d7-70f1474c3c9f","Type":"ContainerStarted","Data":"9472ee5955ca5a2b3ac7bfcd49677bc482125e3434d2cb3c61dee8b50e4969cf"} Oct 30 19:43:38.126270 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:38.125946 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-788b6c6b4d-59ggs" event={"ID":"a179efae-1f42-4b73-b2d7-70f1474c3c9f","Type":"ContainerStarted","Data":"7623105fb7e3da7709ac11895d4789931ba8a97eadfe5e42f7fe5dbb400b225e"} Oct 30 19:43:38.126270 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:38.126066 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-788b6c6b4d-59ggs" Oct 30 19:43:38.237705 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:38.237655 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-788b6c6b4d-59ggs" podStartSLOduration=1.237639336 podStartE2EDuration="1.237639336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:38.236747719 +0000 UTC m=+1121.755063216" watchObservedRunningTime="2026-10-30 19:43:38.237639336 +0000 UTC m=+1121.755954814" Oct 30 19:43:43.222226 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:43:43.222141 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:43:44.871967 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:44.871931 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t"] Oct 30 19:43:44.875618 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:44.875595 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t" Oct 30 19:43:44.925944 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:44.925911 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-4ec609-openshift-default-dockercfg-4txpk\"" Oct 30 19:43:44.933541 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:44.933508 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50-istio-data\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t\" (UID: \"5de2e40c-7f13-4b3c-ab43-5f3a5c351d50\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t" Oct 30 19:43:44.933541 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:44.933539 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50-istio-podinfo\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t\" (UID: \"5de2e40c-7f13-4b3c-ab43-5f3a5c351d50\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t" Oct 30 19:43:44.933738 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:44.933557 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50-credential-socket\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t\" (UID: \"5de2e40c-7f13-4b3c-ab43-5f3a5c351d50\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t" Oct 30 19:43:44.933738 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:44.933718 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50-workload-socket\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t\" (UID: \"5de2e40c-7f13-4b3c-ab43-5f3a5c351d50\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t" Oct 30 19:43:44.933847 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:44.933784 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50-workload-certs\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t\" (UID: \"5de2e40c-7f13-4b3c-ab43-5f3a5c351d50\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t" Oct 30 19:43:44.933901 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:44.933843 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50-istio-envoy\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t\" (UID: \"5de2e40c-7f13-4b3c-ab43-5f3a5c351d50\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t" Oct 30 19:43:44.933901 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:44.933874 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50-istio-token\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t\" (UID: \"5de2e40c-7f13-4b3c-ab43-5f3a5c351d50\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t" Oct 30 19:43:44.934039 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:44.933911 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50-istiod-ca-cert\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t\" (UID: \"5de2e40c-7f13-4b3c-ab43-5f3a5c351d50\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t" Oct 30 19:43:44.934039 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:44.933989 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4ss9\" (UniqueName: \"kubernetes.io/projected/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50-kube-api-access-b4ss9\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t\" (UID: \"5de2e40c-7f13-4b3c-ab43-5f3a5c351d50\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t" Oct 30 19:43:44.974332 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:44.974301 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t"] Oct 30 19:43:45.034439 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:45.034398 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50-istio-envoy\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t\" (UID: \"5de2e40c-7f13-4b3c-ab43-5f3a5c351d50\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t" Oct 30 19:43:45.034439 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:45.034444 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50-istio-token\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t\" (UID: \"5de2e40c-7f13-4b3c-ab43-5f3a5c351d50\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t" Oct 30 19:43:45.034698 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:45.034468 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50-istiod-ca-cert\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t\" (UID: \"5de2e40c-7f13-4b3c-ab43-5f3a5c351d50\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t" Oct 30 19:43:45.034698 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:45.034500 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4ss9\" (UniqueName: \"kubernetes.io/projected/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50-kube-api-access-b4ss9\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t\" (UID: \"5de2e40c-7f13-4b3c-ab43-5f3a5c351d50\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t" Oct 30 19:43:45.034698 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:45.034536 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50-istio-data\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t\" (UID: \"5de2e40c-7f13-4b3c-ab43-5f3a5c351d50\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t" Oct 30 19:43:45.034698 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:45.034571 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50-istio-podinfo\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t\" (UID: \"5de2e40c-7f13-4b3c-ab43-5f3a5c351d50\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t" Oct 30 19:43:45.034923 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:45.034813 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50-credential-socket\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t\" (UID: \"5de2e40c-7f13-4b3c-ab43-5f3a5c351d50\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t" Oct 30 19:43:45.034923 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:45.034911 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50-workload-socket\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t\" (UID: \"5de2e40c-7f13-4b3c-ab43-5f3a5c351d50\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t" Oct 30 19:43:45.034923 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:45.034916 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50-istio-data\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t\" (UID: \"5de2e40c-7f13-4b3c-ab43-5f3a5c351d50\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t" Oct 30 19:43:45.035082 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:45.034980 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50-workload-certs\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t\" (UID: \"5de2e40c-7f13-4b3c-ab43-5f3a5c351d50\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t" Oct 30 19:43:45.035245 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:45.035218 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50-workload-socket\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t\" (UID: \"5de2e40c-7f13-4b3c-ab43-5f3a5c351d50\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t" Oct 30 19:43:45.035529 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:45.035285 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50-workload-certs\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t\" (UID: \"5de2e40c-7f13-4b3c-ab43-5f3a5c351d50\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t" Oct 30 19:43:45.035630 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:45.035536 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50-credential-socket\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t\" (UID: \"5de2e40c-7f13-4b3c-ab43-5f3a5c351d50\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t" Oct 30 19:43:45.035630 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:45.035557 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50-istiod-ca-cert\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t\" (UID: \"5de2e40c-7f13-4b3c-ab43-5f3a5c351d50\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t" Oct 30 19:43:45.037184 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:45.037163 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50-istio-envoy\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t\" (UID: \"5de2e40c-7f13-4b3c-ab43-5f3a5c351d50\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t" Oct 30 19:43:45.037388 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:45.037340 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50-istio-podinfo\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t\" (UID: \"5de2e40c-7f13-4b3c-ab43-5f3a5c351d50\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t" Oct 30 19:43:45.092907 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:45.092874 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50-istio-token\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t\" (UID: \"5de2e40c-7f13-4b3c-ab43-5f3a5c351d50\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t" Oct 30 19:43:45.093189 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:45.093156 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4ss9\" (UniqueName: \"kubernetes.io/projected/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50-kube-api-access-b4ss9\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t\" (UID: \"5de2e40c-7f13-4b3c-ab43-5f3a5c351d50\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t" Oct 30 19:43:45.188848 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:45.188776 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t" Oct 30 19:43:45.603000 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:45.602968 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t"] Oct 30 19:43:45.604857 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:43:45.604828 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5de2e40c_7f13_4b3c_ab43_5f3a5c351d50.slice/crio-cafa8802e6ad0edeb44b76d76255b44d4c6111371929f5f6c085e79b4335cd02 WatchSource:0}: Error finding container cafa8802e6ad0edeb44b76d76255b44d4c6111371929f5f6c085e79b4335cd02: Status 404 returned error can't find the container with id cafa8802e6ad0edeb44b76d76255b44d4c6111371929f5f6c085e79b4335cd02 Oct 30 19:43:45.607094 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:45.607062 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:43:45.607164 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:45.607131 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:43:45.607216 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:45.607183 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:43:46.176763 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:46.176722 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t" event={"ID":"5de2e40c-7f13-4b3c-ab43-5f3a5c351d50","Type":"ContainerStarted","Data":"cd12ef940af7322aff818a272955ff70b4b9fd4d11fef6a7e156416f34129173"} Oct 30 19:43:46.177285 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:46.176787 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t" event={"ID":"5de2e40c-7f13-4b3c-ab43-5f3a5c351d50","Type":"ContainerStarted","Data":"cafa8802e6ad0edeb44b76d76255b44d4c6111371929f5f6c085e79b4335cd02"} Oct 30 19:43:46.189196 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:46.189167 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t" Oct 30 19:43:46.190855 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:46.190832 4292 patch_prober.go:29] interesting pod/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 30 19:43:46.190949 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:46.190876 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t" podUID="5de2e40c-7f13-4b3c-ab43-5f3a5c351d50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 30 19:43:46.322419 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:46.322367 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t" podStartSLOduration=2.322334982 podStartE2EDuration="2.322334982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:46.29288906 +0000 UTC m=+1129.811204541" watchObservedRunningTime="2026-10-30 19:43:46.322334982 +0000 UTC m=+1129.840650439" Oct 30 19:43:47.189379 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:47.189332 4292 patch_prober.go:29] interesting pod/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 30 19:43:47.189769 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:47.189398 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t" podUID="5de2e40c-7f13-4b3c-ab43-5f3a5c351d50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 30 19:43:48.195290 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:48.195256 4292 patch_prober.go:29] interesting pod/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 30 19:43:48.195780 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:48.195316 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t" podUID="5de2e40c-7f13-4b3c-ab43-5f3a5c351d50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 30 19:43:49.135025 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:49.134986 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-788b6c6b4d-59ggs" Oct 30 19:43:49.190024 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:49.189983 4292 patch_prober.go:29] interesting pod/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 30 19:43:49.190203 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:49.190041 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t" podUID="5de2e40c-7f13-4b3c-ab43-5f3a5c351d50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 30 19:43:50.189223 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:50.189186 4292 patch_prober.go:29] interesting pod/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 30 19:43:50.189664 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:50.189250 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t" podUID="5de2e40c-7f13-4b3c-ab43-5f3a5c351d50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 30 19:43:51.189550 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:51.189508 4292 patch_prober.go:29] interesting pod/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 30 19:43:51.190153 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:51.190118 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t" podUID="5de2e40c-7f13-4b3c-ab43-5f3a5c351d50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 30 19:43:52.374669 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:52.374636 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t" Oct 30 19:43:52.376023 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:52.375992 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t" Oct 30 19:43:52.377717 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:52.377680 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t" Oct 30 19:43:55.493605 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:55.493564 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-b0a183/e2e-conflict-a-c338110a-kserve-c7dc6d695-7v6hl"] Oct 30 19:43:55.497056 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:55.497036 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b0a183/e2e-conflict-a-c338110a-kserve-c7dc6d695-7v6hl" Oct 30 19:43:55.546719 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:55.546693 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-b0a183\"/\"kube-root-ca.crt\"" Oct 30 19:43:55.566326 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:55.566277 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc2aacd3-0456-4c2b-8d5e-3743ec9d872e-dshm\") pod \"e2e-conflict-a-c338110a-kserve-c7dc6d695-7v6hl\" (UID: \"cc2aacd3-0456-4c2b-8d5e-3743ec9d872e\") " pod="e2e-models-e2e-worker-w2-b0a183/e2e-conflict-a-c338110a-kserve-c7dc6d695-7v6hl" Oct 30 19:43:55.566534 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:55.566510 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc2aacd3-0456-4c2b-8d5e-3743ec9d872e-tmp-dir\") pod \"e2e-conflict-a-c338110a-kserve-c7dc6d695-7v6hl\" (UID: \"cc2aacd3-0456-4c2b-8d5e-3743ec9d872e\") " pod="e2e-models-e2e-worker-w2-b0a183/e2e-conflict-a-c338110a-kserve-c7dc6d695-7v6hl" Oct 30 19:43:55.566582 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:55.566554 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctpqp\" (UniqueName: \"kubernetes.io/projected/cc2aacd3-0456-4c2b-8d5e-3743ec9d872e-kube-api-access-ctpqp\") pod \"e2e-conflict-a-c338110a-kserve-c7dc6d695-7v6hl\" (UID: \"cc2aacd3-0456-4c2b-8d5e-3743ec9d872e\") " pod="e2e-models-e2e-worker-w2-b0a183/e2e-conflict-a-c338110a-kserve-c7dc6d695-7v6hl" Oct 30 19:43:55.566698 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:55.566675 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc2aacd3-0456-4c2b-8d5e-3743ec9d872e-tls-certs\") pod \"e2e-conflict-a-c338110a-kserve-c7dc6d695-7v6hl\" (UID: \"cc2aacd3-0456-4c2b-8d5e-3743ec9d872e\") " pod="e2e-models-e2e-worker-w2-b0a183/e2e-conflict-a-c338110a-kserve-c7dc6d695-7v6hl" Oct 30 19:43:55.566817 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:55.566796 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc2aacd3-0456-4c2b-8d5e-3743ec9d872e-model-cache\") pod \"e2e-conflict-a-c338110a-kserve-c7dc6d695-7v6hl\" (UID: \"cc2aacd3-0456-4c2b-8d5e-3743ec9d872e\") " pod="e2e-models-e2e-worker-w2-b0a183/e2e-conflict-a-c338110a-kserve-c7dc6d695-7v6hl" Oct 30 19:43:55.566878 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:55.566857 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc2aacd3-0456-4c2b-8d5e-3743ec9d872e-home\") pod \"e2e-conflict-a-c338110a-kserve-c7dc6d695-7v6hl\" (UID: \"cc2aacd3-0456-4c2b-8d5e-3743ec9d872e\") " pod="e2e-models-e2e-worker-w2-b0a183/e2e-conflict-a-c338110a-kserve-c7dc6d695-7v6hl" Oct 30 19:43:55.593518 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:55.593470 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-b0a183\"/\"openshift-service-ca.crt\"" Oct 30 19:43:55.593725 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:55.593596 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-b0a183\"/\"default-dockercfg-xjrrw\"" Oct 30 19:43:55.593725 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:55.593641 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-b0a183\"/\"e2e-conflict-a-c338110a-kserve-self-signed-certs\"" Oct 30 19:43:55.595770 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:55.595743 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-b0a183/e2e-conflict-a-c338110a-kserve-c7dc6d695-7v6hl"] Oct 30 19:43:55.668255 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:55.668208 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc2aacd3-0456-4c2b-8d5e-3743ec9d872e-tmp-dir\") pod \"e2e-conflict-a-c338110a-kserve-c7dc6d695-7v6hl\" (UID: \"cc2aacd3-0456-4c2b-8d5e-3743ec9d872e\") " pod="e2e-models-e2e-worker-w2-b0a183/e2e-conflict-a-c338110a-kserve-c7dc6d695-7v6hl" Oct 30 19:43:55.668255 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:55.668266 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctpqp\" (UniqueName: \"kubernetes.io/projected/cc2aacd3-0456-4c2b-8d5e-3743ec9d872e-kube-api-access-ctpqp\") pod \"e2e-conflict-a-c338110a-kserve-c7dc6d695-7v6hl\" (UID: \"cc2aacd3-0456-4c2b-8d5e-3743ec9d872e\") " pod="e2e-models-e2e-worker-w2-b0a183/e2e-conflict-a-c338110a-kserve-c7dc6d695-7v6hl" Oct 30 19:43:55.668568 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:55.668292 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc2aacd3-0456-4c2b-8d5e-3743ec9d872e-tls-certs\") pod \"e2e-conflict-a-c338110a-kserve-c7dc6d695-7v6hl\" (UID: \"cc2aacd3-0456-4c2b-8d5e-3743ec9d872e\") " pod="e2e-models-e2e-worker-w2-b0a183/e2e-conflict-a-c338110a-kserve-c7dc6d695-7v6hl" Oct 30 19:43:55.668568 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:55.668393 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc2aacd3-0456-4c2b-8d5e-3743ec9d872e-model-cache\") pod \"e2e-conflict-a-c338110a-kserve-c7dc6d695-7v6hl\" (UID: \"cc2aacd3-0456-4c2b-8d5e-3743ec9d872e\") " pod="e2e-models-e2e-worker-w2-b0a183/e2e-conflict-a-c338110a-kserve-c7dc6d695-7v6hl" Oct 30 19:43:55.668568 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:55.668420 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc2aacd3-0456-4c2b-8d5e-3743ec9d872e-home\") pod \"e2e-conflict-a-c338110a-kserve-c7dc6d695-7v6hl\" (UID: \"cc2aacd3-0456-4c2b-8d5e-3743ec9d872e\") " pod="e2e-models-e2e-worker-w2-b0a183/e2e-conflict-a-c338110a-kserve-c7dc6d695-7v6hl" Oct 30 19:43:55.668568 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:55.668449 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc2aacd3-0456-4c2b-8d5e-3743ec9d872e-dshm\") pod \"e2e-conflict-a-c338110a-kserve-c7dc6d695-7v6hl\" (UID: \"cc2aacd3-0456-4c2b-8d5e-3743ec9d872e\") " pod="e2e-models-e2e-worker-w2-b0a183/e2e-conflict-a-c338110a-kserve-c7dc6d695-7v6hl" Oct 30 19:43:55.669189 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:55.669157 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc2aacd3-0456-4c2b-8d5e-3743ec9d872e-tmp-dir\") pod \"e2e-conflict-a-c338110a-kserve-c7dc6d695-7v6hl\" (UID: \"cc2aacd3-0456-4c2b-8d5e-3743ec9d872e\") " pod="e2e-models-e2e-worker-w2-b0a183/e2e-conflict-a-c338110a-kserve-c7dc6d695-7v6hl" Oct 30 19:43:55.669405 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:55.669308 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc2aacd3-0456-4c2b-8d5e-3743ec9d872e-home\") pod \"e2e-conflict-a-c338110a-kserve-c7dc6d695-7v6hl\" (UID: \"cc2aacd3-0456-4c2b-8d5e-3743ec9d872e\") " pod="e2e-models-e2e-worker-w2-b0a183/e2e-conflict-a-c338110a-kserve-c7dc6d695-7v6hl" Oct 30 19:43:55.669538 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:55.669438 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc2aacd3-0456-4c2b-8d5e-3743ec9d872e-model-cache\") pod \"e2e-conflict-a-c338110a-kserve-c7dc6d695-7v6hl\" (UID: \"cc2aacd3-0456-4c2b-8d5e-3743ec9d872e\") " pod="e2e-models-e2e-worker-w2-b0a183/e2e-conflict-a-c338110a-kserve-c7dc6d695-7v6hl" Oct 30 19:43:55.672004 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:55.671978 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc2aacd3-0456-4c2b-8d5e-3743ec9d872e-dshm\") pod \"e2e-conflict-a-c338110a-kserve-c7dc6d695-7v6hl\" (UID: \"cc2aacd3-0456-4c2b-8d5e-3743ec9d872e\") " pod="e2e-models-e2e-worker-w2-b0a183/e2e-conflict-a-c338110a-kserve-c7dc6d695-7v6hl" Oct 30 19:43:55.673741 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:55.673700 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc2aacd3-0456-4c2b-8d5e-3743ec9d872e-tls-certs\") pod \"e2e-conflict-a-c338110a-kserve-c7dc6d695-7v6hl\" (UID: \"cc2aacd3-0456-4c2b-8d5e-3743ec9d872e\") " pod="e2e-models-e2e-worker-w2-b0a183/e2e-conflict-a-c338110a-kserve-c7dc6d695-7v6hl" Oct 30 19:43:55.728338 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:55.728295 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctpqp\" (UniqueName: \"kubernetes.io/projected/cc2aacd3-0456-4c2b-8d5e-3743ec9d872e-kube-api-access-ctpqp\") pod \"e2e-conflict-a-c338110a-kserve-c7dc6d695-7v6hl\" (UID: \"cc2aacd3-0456-4c2b-8d5e-3743ec9d872e\") " pod="e2e-models-e2e-worker-w2-b0a183/e2e-conflict-a-c338110a-kserve-c7dc6d695-7v6hl" Oct 30 19:43:55.809308 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:55.809275 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b0a183/e2e-conflict-a-c338110a-kserve-c7dc6d695-7v6hl" Oct 30 19:43:56.239754 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:56.239731 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-b0a183/e2e-conflict-a-c338110a-kserve-c7dc6d695-7v6hl"] Oct 30 19:43:56.241518 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:43:56.241490 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc2aacd3_0456_4c2b_8d5e_3743ec9d872e.slice/crio-ca67dc9952c86a8b2c1d10f275a635d170c25882d877ec5f5eaceb9aa7841376 WatchSource:0}: Error finding container ca67dc9952c86a8b2c1d10f275a635d170c25882d877ec5f5eaceb9aa7841376: Status 404 returned error can't find the container with id ca67dc9952c86a8b2c1d10f275a635d170c25882d877ec5f5eaceb9aa7841376 Oct 30 19:43:57.232733 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:57.232694 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b0a183/e2e-conflict-a-c338110a-kserve-c7dc6d695-7v6hl" event={"ID":"cc2aacd3-0456-4c2b-8d5e-3743ec9d872e","Type":"ContainerStarted","Data":"0cb68409f2e2a127f7fbb1542bd6e7fd0a8da215235f9425b84c72579f1f63d3"} Oct 30 19:43:57.232733 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:57.232743 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b0a183/e2e-conflict-a-c338110a-kserve-c7dc6d695-7v6hl" event={"ID":"cc2aacd3-0456-4c2b-8d5e-3743ec9d872e","Type":"ContainerStarted","Data":"ca67dc9952c86a8b2c1d10f275a635d170c25882d877ec5f5eaceb9aa7841376"} Oct 30 19:43:57.345940 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:57.345878 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-b0a183/e2e-conflict-a-c338110a-kserve-c7dc6d695-7v6hl" podStartSLOduration=2.345861606 podStartE2EDuration="2.345861606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:57.343343624 +0000 UTC m=+1140.861659081" watchObservedRunningTime="2026-10-30 19:43:57.345861606 +0000 UTC m=+1140.864177116" Oct 30 19:43:57.809592 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:43:57.809557 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b0a183/e2e-conflict-a-c338110a-kserve-c7dc6d695-7v6hl" Oct 30 19:44:05.809920 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:05.809877 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-b0a183/e2e-conflict-a-c338110a-kserve-c7dc6d695-7v6hl" Oct 30 19:44:05.824511 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:05.824484 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-b0a183/e2e-conflict-a-c338110a-kserve-c7dc6d695-7v6hl" Oct 30 19:44:06.292842 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:06.292802 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b0a183/e2e-conflict-a-c338110a-kserve-c7dc6d695-7v6hl" Oct 30 19:44:13.222458 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:44:13.222410 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:44:19.297885 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:19.297849 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lvnj7"] Oct 30 19:44:19.302004 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:19.301974 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lvnj7" Oct 30 19:44:19.420429 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:19.420388 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-lvnj7\" (UID: \"317c18b6-5b6b-4af5-9ba3-3d5ad719d53b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lvnj7" Oct 30 19:44:19.420429 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:19.420430 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-lvnj7\" (UID: \"317c18b6-5b6b-4af5-9ba3-3d5ad719d53b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lvnj7" Oct 30 19:44:19.420700 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:19.420541 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-lvnj7\" (UID: \"317c18b6-5b6b-4af5-9ba3-3d5ad719d53b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lvnj7" Oct 30 19:44:19.420700 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:19.420576 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-lvnj7\" (UID: \"317c18b6-5b6b-4af5-9ba3-3d5ad719d53b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lvnj7" Oct 30 19:44:19.420700 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:19.420669 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-lvnj7\" (UID: \"317c18b6-5b6b-4af5-9ba3-3d5ad719d53b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lvnj7" Oct 30 19:44:19.421101 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:19.420704 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-lvnj7\" (UID: \"317c18b6-5b6b-4af5-9ba3-3d5ad719d53b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lvnj7" Oct 30 19:44:19.421101 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:19.420744 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-lvnj7\" (UID: \"317c18b6-5b6b-4af5-9ba3-3d5ad719d53b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lvnj7" Oct 30 19:44:19.421101 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:19.420821 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-lvnj7\" (UID: \"317c18b6-5b6b-4af5-9ba3-3d5ad719d53b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lvnj7" Oct 30 19:44:19.421101 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:19.420852 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qjxp\" (UniqueName: \"kubernetes.io/projected/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b-kube-api-access-6qjxp\") pod \"maas-default-gateway-openshift-default-694db55c87-lvnj7\" (UID: \"317c18b6-5b6b-4af5-9ba3-3d5ad719d53b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lvnj7" Oct 30 19:44:19.499095 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:19.499056 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lvnj7"] Oct 30 19:44:19.522455 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:19.522412 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-lvnj7\" (UID: \"317c18b6-5b6b-4af5-9ba3-3d5ad719d53b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lvnj7" Oct 30 19:44:19.522624 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:19.522522 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-lvnj7\" (UID: \"317c18b6-5b6b-4af5-9ba3-3d5ad719d53b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lvnj7" Oct 30 19:44:19.522624 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:19.522578 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qjxp\" (UniqueName: \"kubernetes.io/projected/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b-kube-api-access-6qjxp\") pod \"maas-default-gateway-openshift-default-694db55c87-lvnj7\" (UID: \"317c18b6-5b6b-4af5-9ba3-3d5ad719d53b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lvnj7" Oct 30 19:44:19.522718 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:19.522645 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-lvnj7\" (UID: \"317c18b6-5b6b-4af5-9ba3-3d5ad719d53b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lvnj7" Oct 30 19:44:19.522718 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:19.522672 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-lvnj7\" (UID: \"317c18b6-5b6b-4af5-9ba3-3d5ad719d53b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lvnj7" Oct 30 19:44:19.522830 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:19.522724 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-lvnj7\" (UID: \"317c18b6-5b6b-4af5-9ba3-3d5ad719d53b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lvnj7" Oct 30 19:44:19.522830 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:19.522757 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-lvnj7\" (UID: \"317c18b6-5b6b-4af5-9ba3-3d5ad719d53b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lvnj7" Oct 30 19:44:19.522830 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:19.522812 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-lvnj7\" (UID: \"317c18b6-5b6b-4af5-9ba3-3d5ad719d53b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lvnj7" Oct 30 19:44:19.523075 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:19.522839 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-lvnj7\" (UID: \"317c18b6-5b6b-4af5-9ba3-3d5ad719d53b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lvnj7" Oct 30 19:44:19.523075 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:19.522847 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-lvnj7\" (UID: \"317c18b6-5b6b-4af5-9ba3-3d5ad719d53b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lvnj7" Oct 30 19:44:19.523075 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:19.522919 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-lvnj7\" (UID: \"317c18b6-5b6b-4af5-9ba3-3d5ad719d53b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lvnj7" Oct 30 19:44:19.523395 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:19.523317 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-lvnj7\" (UID: \"317c18b6-5b6b-4af5-9ba3-3d5ad719d53b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lvnj7" Oct 30 19:44:19.523617 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:19.523592 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-lvnj7\" (UID: \"317c18b6-5b6b-4af5-9ba3-3d5ad719d53b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lvnj7" Oct 30 19:44:19.523719 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:19.523647 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-lvnj7\" (UID: \"317c18b6-5b6b-4af5-9ba3-3d5ad719d53b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lvnj7" Oct 30 19:44:19.525425 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:19.525400 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-lvnj7\" (UID: \"317c18b6-5b6b-4af5-9ba3-3d5ad719d53b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lvnj7" Oct 30 19:44:19.525537 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:19.525519 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-lvnj7\" (UID: \"317c18b6-5b6b-4af5-9ba3-3d5ad719d53b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lvnj7" Oct 30 19:44:19.581721 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:19.581647 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qjxp\" (UniqueName: \"kubernetes.io/projected/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b-kube-api-access-6qjxp\") pod \"maas-default-gateway-openshift-default-694db55c87-lvnj7\" (UID: \"317c18b6-5b6b-4af5-9ba3-3d5ad719d53b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lvnj7" Oct 30 19:44:19.581721 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:19.581661 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-lvnj7\" (UID: \"317c18b6-5b6b-4af5-9ba3-3d5ad719d53b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lvnj7" Oct 30 19:44:19.614602 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:19.614568 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lvnj7" Oct 30 19:44:20.032473 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:20.032403 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lvnj7"] Oct 30 19:44:20.034844 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:44:20.034815 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod317c18b6_5b6b_4af5_9ba3_3d5ad719d53b.slice/crio-b3363d40674ccadc3cd8513dcd9d50b8281205aa3adf792267e8ee692e851422 WatchSource:0}: Error finding container b3363d40674ccadc3cd8513dcd9d50b8281205aa3adf792267e8ee692e851422: Status 404 returned error can't find the container with id b3363d40674ccadc3cd8513dcd9d50b8281205aa3adf792267e8ee692e851422 Oct 30 19:44:20.037058 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:20.037026 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:44:20.037152 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:20.037093 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:44:20.037152 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:20.037121 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:44:20.345902 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:20.345809 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lvnj7" event={"ID":"317c18b6-5b6b-4af5-9ba3-3d5ad719d53b","Type":"ContainerStarted","Data":"0d0c4377197511e139ad5eb1f0542269cc672c1912d7409d867069d1ff3ff768"} Oct 30 19:44:20.345902 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:20.345865 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lvnj7" event={"ID":"317c18b6-5b6b-4af5-9ba3-3d5ad719d53b","Type":"ContainerStarted","Data":"b3363d40674ccadc3cd8513dcd9d50b8281205aa3adf792267e8ee692e851422"} Oct 30 19:44:20.463944 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:20.463878 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lvnj7" podStartSLOduration=1.463859315 podStartE2EDuration="1.463859315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:20.460343065 +0000 UTC m=+1163.978658546" watchObservedRunningTime="2026-10-30 19:44:20.463859315 +0000 UTC m=+1163.982174773" Oct 30 19:44:20.614826 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:20.614755 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lvnj7" Oct 30 19:44:21.616773 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:21.616733 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-lvnj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Oct 30 19:44:21.617179 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:21.616793 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lvnj7" podUID="317c18b6-5b6b-4af5-9ba3-3d5ad719d53b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Oct 30 19:44:22.615003 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:22.614968 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-lvnj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Oct 30 19:44:22.615188 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:22.615022 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lvnj7" podUID="317c18b6-5b6b-4af5-9ba3-3d5ad719d53b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Oct 30 19:44:23.625659 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:23.623609 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-lvnj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Oct 30 19:44:23.625659 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:23.623676 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lvnj7" podUID="317c18b6-5b6b-4af5-9ba3-3d5ad719d53b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Oct 30 19:44:24.615064 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:24.615032 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-lvnj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Oct 30 19:44:24.615258 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:24.615088 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lvnj7" podUID="317c18b6-5b6b-4af5-9ba3-3d5ad719d53b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Oct 30 19:44:25.615178 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:25.615139 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-lvnj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Oct 30 19:44:25.615705 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:25.615198 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lvnj7" podUID="317c18b6-5b6b-4af5-9ba3-3d5ad719d53b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Oct 30 19:44:26.615148 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:26.615113 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-lvnj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Oct 30 19:44:26.615336 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:26.615172 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lvnj7" podUID="317c18b6-5b6b-4af5-9ba3-3d5ad719d53b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Oct 30 19:44:27.614940 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:27.614900 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-lvnj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Oct 30 19:44:27.615134 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:27.614958 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lvnj7" podUID="317c18b6-5b6b-4af5-9ba3-3d5ad719d53b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Oct 30 19:44:28.619286 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:28.619254 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lvnj7" Oct 30 19:44:28.619705 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:28.619557 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lvnj7" Oct 30 19:44:28.620146 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:28.620126 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lvnj7" Oct 30 19:44:31.739941 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:31.739906 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9"] Oct 30 19:44:31.744047 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:31.744022 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9" Oct 30 19:44:31.784654 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:31.784621 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxfb9\" (UniqueName: \"kubernetes.io/projected/4f2b3150-1217-44bc-976a-65c86263223a-kube-api-access-hxfb9\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9\" (UID: \"4f2b3150-1217-44bc-976a-65c86263223a\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9" Oct 30 19:44:31.784654 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:31.784663 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f2b3150-1217-44bc-976a-65c86263223a-istio-data\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9\" (UID: \"4f2b3150-1217-44bc-976a-65c86263223a\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9" Oct 30 19:44:31.784920 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:31.784728 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f2b3150-1217-44bc-976a-65c86263223a-credential-socket\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9\" (UID: \"4f2b3150-1217-44bc-976a-65c86263223a\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9" Oct 30 19:44:31.784920 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:31.784810 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f2b3150-1217-44bc-976a-65c86263223a-workload-certs\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9\" (UID: \"4f2b3150-1217-44bc-976a-65c86263223a\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9" Oct 30 19:44:31.784920 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:31.784840 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f2b3150-1217-44bc-976a-65c86263223a-istio-podinfo\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9\" (UID: \"4f2b3150-1217-44bc-976a-65c86263223a\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9" Oct 30 19:44:31.784920 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:31.784870 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f2b3150-1217-44bc-976a-65c86263223a-workload-socket\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9\" (UID: \"4f2b3150-1217-44bc-976a-65c86263223a\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9" Oct 30 19:44:31.785149 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:31.784982 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f2b3150-1217-44bc-976a-65c86263223a-istiod-ca-cert\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9\" (UID: \"4f2b3150-1217-44bc-976a-65c86263223a\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9" Oct 30 19:44:31.785149 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:31.785012 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f2b3150-1217-44bc-976a-65c86263223a-istio-token\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9\" (UID: \"4f2b3150-1217-44bc-976a-65c86263223a\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9" Oct 30 19:44:31.785149 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:31.785100 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f2b3150-1217-44bc-976a-65c86263223a-istio-envoy\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9\" (UID: \"4f2b3150-1217-44bc-976a-65c86263223a\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9" Oct 30 19:44:31.793869 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:31.793848 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-b0a183-openshift-default-dockercfg-hjv6k\"" Oct 30 19:44:31.842652 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:31.842618 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9"] Oct 30 19:44:31.885793 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:31.885763 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f2b3150-1217-44bc-976a-65c86263223a-istio-podinfo\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9\" (UID: \"4f2b3150-1217-44bc-976a-65c86263223a\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9" Oct 30 19:44:31.886001 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:31.885809 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f2b3150-1217-44bc-976a-65c86263223a-workload-socket\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9\" (UID: \"4f2b3150-1217-44bc-976a-65c86263223a\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9" Oct 30 19:44:31.886001 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:31.885830 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f2b3150-1217-44bc-976a-65c86263223a-istiod-ca-cert\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9\" (UID: \"4f2b3150-1217-44bc-976a-65c86263223a\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9" Oct 30 19:44:31.886001 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:31.885939 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f2b3150-1217-44bc-976a-65c86263223a-istio-token\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9\" (UID: \"4f2b3150-1217-44bc-976a-65c86263223a\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9" Oct 30 19:44:31.886001 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:31.885985 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f2b3150-1217-44bc-976a-65c86263223a-istio-envoy\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9\" (UID: \"4f2b3150-1217-44bc-976a-65c86263223a\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9" Oct 30 19:44:31.886231 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:31.886057 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxfb9\" (UniqueName: \"kubernetes.io/projected/4f2b3150-1217-44bc-976a-65c86263223a-kube-api-access-hxfb9\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9\" (UID: \"4f2b3150-1217-44bc-976a-65c86263223a\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9" Oct 30 19:44:31.886231 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:31.886080 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f2b3150-1217-44bc-976a-65c86263223a-istio-data\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9\" (UID: \"4f2b3150-1217-44bc-976a-65c86263223a\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9" Oct 30 19:44:31.886340 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:31.886274 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f2b3150-1217-44bc-976a-65c86263223a-workload-socket\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9\" (UID: \"4f2b3150-1217-44bc-976a-65c86263223a\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9" Oct 30 19:44:31.886410 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:31.886379 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f2b3150-1217-44bc-976a-65c86263223a-credential-socket\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9\" (UID: \"4f2b3150-1217-44bc-976a-65c86263223a\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9" Oct 30 19:44:31.886492 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:31.886469 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f2b3150-1217-44bc-976a-65c86263223a-istio-data\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9\" (UID: \"4f2b3150-1217-44bc-976a-65c86263223a\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9" Oct 30 19:44:31.886556 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:31.886489 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f2b3150-1217-44bc-976a-65c86263223a-workload-certs\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9\" (UID: \"4f2b3150-1217-44bc-976a-65c86263223a\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9" Oct 30 19:44:31.886645 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:31.886626 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f2b3150-1217-44bc-976a-65c86263223a-istiod-ca-cert\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9\" (UID: \"4f2b3150-1217-44bc-976a-65c86263223a\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9" Oct 30 19:44:31.886760 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:31.886742 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f2b3150-1217-44bc-976a-65c86263223a-credential-socket\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9\" (UID: \"4f2b3150-1217-44bc-976a-65c86263223a\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9" Oct 30 19:44:31.886848 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:31.886828 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f2b3150-1217-44bc-976a-65c86263223a-workload-certs\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9\" (UID: \"4f2b3150-1217-44bc-976a-65c86263223a\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9" Oct 30 19:44:31.888343 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:31.888319 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f2b3150-1217-44bc-976a-65c86263223a-istio-envoy\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9\" (UID: \"4f2b3150-1217-44bc-976a-65c86263223a\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9" Oct 30 19:44:31.888589 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:31.888569 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f2b3150-1217-44bc-976a-65c86263223a-istio-podinfo\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9\" (UID: \"4f2b3150-1217-44bc-976a-65c86263223a\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9" Oct 30 19:44:31.944401 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:31.944347 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f2b3150-1217-44bc-976a-65c86263223a-istio-token\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9\" (UID: \"4f2b3150-1217-44bc-976a-65c86263223a\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9" Oct 30 19:44:31.944546 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:31.944375 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxfb9\" (UniqueName: \"kubernetes.io/projected/4f2b3150-1217-44bc-976a-65c86263223a-kube-api-access-hxfb9\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9\" (UID: \"4f2b3150-1217-44bc-976a-65c86263223a\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9" Oct 30 19:44:32.056916 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:32.056883 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9" Oct 30 19:44:32.477492 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:32.477469 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9"] Oct 30 19:44:32.479286 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:44:32.479259 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f2b3150_1217_44bc_976a_65c86263223a.slice/crio-d6154a0575fdc6ed4f1e178272df401c0f0c5dccd746ca68664b0c41972a2b49 WatchSource:0}: Error finding container d6154a0575fdc6ed4f1e178272df401c0f0c5dccd746ca68664b0c41972a2b49: Status 404 returned error can't find the container with id d6154a0575fdc6ed4f1e178272df401c0f0c5dccd746ca68664b0c41972a2b49 Oct 30 19:44:32.481687 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:32.481656 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:44:32.481766 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:32.481721 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:44:32.481766 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:32.481750 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:44:33.433794 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:33.433759 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9" event={"ID":"4f2b3150-1217-44bc-976a-65c86263223a","Type":"ContainerStarted","Data":"009626f5a7dc245ef9554b5c86581a5211c4fb43ca841e6dc21f5d2b4949e0ba"} Oct 30 19:44:33.434222 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:33.433817 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9" event={"ID":"4f2b3150-1217-44bc-976a-65c86263223a","Type":"ContainerStarted","Data":"d6154a0575fdc6ed4f1e178272df401c0f0c5dccd746ca68664b0c41972a2b49"} Oct 30 19:44:33.642485 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:33.642432 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9" podStartSLOduration=2.642417771 podStartE2EDuration="2.642417771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:33.5951056 +0000 UTC m=+1177.113421077" watchObservedRunningTime="2026-10-30 19:44:33.642417771 +0000 UTC m=+1177.160733227" Oct 30 19:44:34.059430 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:34.059395 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9" Oct 30 19:44:34.059839 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:34.059815 4292 patch_prober.go:29] interesting pod/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Oct 30 19:44:34.059996 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:34.059973 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9" podUID="4f2b3150-1217-44bc-976a-65c86263223a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Oct 30 19:44:34.398130 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:34.398041 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8nrtp"] Oct 30 19:44:34.402042 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:34.402014 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8nrtp" Oct 30 19:44:34.498041 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:34.498004 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8nrtp"] Oct 30 19:44:34.535468 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:34.535425 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5978bc5-3ff8-4f4a-b85f-58b737179fee-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-8nrtp\" (UID: \"b5978bc5-3ff8-4f4a-b85f-58b737179fee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8nrtp" Oct 30 19:44:34.535689 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:34.535578 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5978bc5-3ff8-4f4a-b85f-58b737179fee-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-8nrtp\" (UID: \"b5978bc5-3ff8-4f4a-b85f-58b737179fee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8nrtp" Oct 30 19:44:34.535689 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:34.535639 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tng4\" (UniqueName: \"kubernetes.io/projected/b5978bc5-3ff8-4f4a-b85f-58b737179fee-kube-api-access-6tng4\") pod \"maas-default-gateway-openshift-default-694db55c87-8nrtp\" (UID: \"b5978bc5-3ff8-4f4a-b85f-58b737179fee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8nrtp" Oct 30 19:44:34.535937 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:34.535911 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5978bc5-3ff8-4f4a-b85f-58b737179fee-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-8nrtp\" (UID: \"b5978bc5-3ff8-4f4a-b85f-58b737179fee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8nrtp" Oct 30 19:44:34.536006 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:34.535968 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5978bc5-3ff8-4f4a-b85f-58b737179fee-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-8nrtp\" (UID: \"b5978bc5-3ff8-4f4a-b85f-58b737179fee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8nrtp" Oct 30 19:44:34.536077 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:34.536060 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5978bc5-3ff8-4f4a-b85f-58b737179fee-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-8nrtp\" (UID: \"b5978bc5-3ff8-4f4a-b85f-58b737179fee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8nrtp" Oct 30 19:44:34.536138 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:34.536086 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5978bc5-3ff8-4f4a-b85f-58b737179fee-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-8nrtp\" (UID: \"b5978bc5-3ff8-4f4a-b85f-58b737179fee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8nrtp" Oct 30 19:44:34.536196 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:34.536160 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5978bc5-3ff8-4f4a-b85f-58b737179fee-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-8nrtp\" (UID: \"b5978bc5-3ff8-4f4a-b85f-58b737179fee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8nrtp" Oct 30 19:44:34.536254 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:34.536205 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5978bc5-3ff8-4f4a-b85f-58b737179fee-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-8nrtp\" (UID: \"b5978bc5-3ff8-4f4a-b85f-58b737179fee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8nrtp" Oct 30 19:44:34.637522 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:34.637483 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5978bc5-3ff8-4f4a-b85f-58b737179fee-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-8nrtp\" (UID: \"b5978bc5-3ff8-4f4a-b85f-58b737179fee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8nrtp" Oct 30 19:44:34.637719 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:34.637537 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5978bc5-3ff8-4f4a-b85f-58b737179fee-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-8nrtp\" (UID: \"b5978bc5-3ff8-4f4a-b85f-58b737179fee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8nrtp" Oct 30 19:44:34.637719 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:34.637612 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5978bc5-3ff8-4f4a-b85f-58b737179fee-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-8nrtp\" (UID: \"b5978bc5-3ff8-4f4a-b85f-58b737179fee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8nrtp" Oct 30 19:44:34.637719 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:34.637639 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5978bc5-3ff8-4f4a-b85f-58b737179fee-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-8nrtp\" (UID: \"b5978bc5-3ff8-4f4a-b85f-58b737179fee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8nrtp" Oct 30 19:44:34.637719 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:34.637667 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5978bc5-3ff8-4f4a-b85f-58b737179fee-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-8nrtp\" (UID: \"b5978bc5-3ff8-4f4a-b85f-58b737179fee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8nrtp" Oct 30 19:44:34.637719 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:34.637694 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5978bc5-3ff8-4f4a-b85f-58b737179fee-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-8nrtp\" (UID: \"b5978bc5-3ff8-4f4a-b85f-58b737179fee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8nrtp" Oct 30 19:44:34.637986 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:34.637765 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5978bc5-3ff8-4f4a-b85f-58b737179fee-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-8nrtp\" (UID: \"b5978bc5-3ff8-4f4a-b85f-58b737179fee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8nrtp" Oct 30 19:44:34.637986 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:34.637833 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5978bc5-3ff8-4f4a-b85f-58b737179fee-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-8nrtp\" (UID: \"b5978bc5-3ff8-4f4a-b85f-58b737179fee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8nrtp" Oct 30 19:44:34.637986 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:34.637883 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tng4\" (UniqueName: \"kubernetes.io/projected/b5978bc5-3ff8-4f4a-b85f-58b737179fee-kube-api-access-6tng4\") pod \"maas-default-gateway-openshift-default-694db55c87-8nrtp\" (UID: \"b5978bc5-3ff8-4f4a-b85f-58b737179fee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8nrtp" Oct 30 19:44:34.638132 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:34.638087 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5978bc5-3ff8-4f4a-b85f-58b737179fee-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-8nrtp\" (UID: \"b5978bc5-3ff8-4f4a-b85f-58b737179fee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8nrtp" Oct 30 19:44:34.638196 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:34.638134 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5978bc5-3ff8-4f4a-b85f-58b737179fee-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-8nrtp\" (UID: \"b5978bc5-3ff8-4f4a-b85f-58b737179fee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8nrtp" Oct 30 19:44:34.638447 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:34.638422 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5978bc5-3ff8-4f4a-b85f-58b737179fee-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-8nrtp\" (UID: \"b5978bc5-3ff8-4f4a-b85f-58b737179fee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8nrtp" Oct 30 19:44:34.638554 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:34.638424 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5978bc5-3ff8-4f4a-b85f-58b737179fee-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-8nrtp\" (UID: \"b5978bc5-3ff8-4f4a-b85f-58b737179fee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8nrtp" Oct 30 19:44:34.638554 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:34.638472 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5978bc5-3ff8-4f4a-b85f-58b737179fee-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-8nrtp\" (UID: \"b5978bc5-3ff8-4f4a-b85f-58b737179fee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8nrtp" Oct 30 19:44:34.639793 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:34.639769 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5978bc5-3ff8-4f4a-b85f-58b737179fee-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-8nrtp\" (UID: \"b5978bc5-3ff8-4f4a-b85f-58b737179fee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8nrtp" Oct 30 19:44:34.640180 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:34.640159 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5978bc5-3ff8-4f4a-b85f-58b737179fee-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-8nrtp\" (UID: \"b5978bc5-3ff8-4f4a-b85f-58b737179fee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8nrtp" Oct 30 19:44:34.696838 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:34.696762 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5978bc5-3ff8-4f4a-b85f-58b737179fee-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-8nrtp\" (UID: \"b5978bc5-3ff8-4f4a-b85f-58b737179fee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8nrtp" Oct 30 19:44:34.696999 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:34.696840 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tng4\" (UniqueName: \"kubernetes.io/projected/b5978bc5-3ff8-4f4a-b85f-58b737179fee-kube-api-access-6tng4\") pod \"maas-default-gateway-openshift-default-694db55c87-8nrtp\" (UID: \"b5978bc5-3ff8-4f4a-b85f-58b737179fee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8nrtp" Oct 30 19:44:34.715779 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:34.715745 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8nrtp" Oct 30 19:44:35.059862 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:35.059816 4292 patch_prober.go:29] interesting pod/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Oct 30 19:44:35.060161 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:35.060130 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9" podUID="4f2b3150-1217-44bc-976a-65c86263223a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Oct 30 19:44:35.179870 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:35.179830 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8nrtp"] Oct 30 19:44:35.183274 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:35.183221 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:44:35.183452 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:35.183425 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:44:35.183549 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:35.183528 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:44:35.445448 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:35.445298 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8nrtp" event={"ID":"b5978bc5-3ff8-4f4a-b85f-58b737179fee","Type":"ContainerStarted","Data":"7ae7bbea353c7a21ae7b84d11a3cd62c3b88e4c4a1907e08b9babfb70f3b335f"} Oct 30 19:44:35.445448 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:35.445409 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8nrtp" event={"ID":"b5978bc5-3ff8-4f4a-b85f-58b737179fee","Type":"ContainerStarted","Data":"b27b411ed02fe38282c171739f425f4d9ec1d7fd15888ac44d5d51c4dd7cc25d"} Oct 30 19:44:35.566377 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:35.565194 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8nrtp" podStartSLOduration=1.565175712 podStartE2EDuration="1.565175712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:35.564663982 +0000 UTC m=+1179.082979432" watchObservedRunningTime="2026-10-30 19:44:35.565175712 +0000 UTC m=+1179.083491173" Oct 30 19:44:35.716566 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:35.716478 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8nrtp" Oct 30 19:44:36.063378 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:36.057830 4292 patch_prober.go:29] interesting pod/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Oct 30 19:44:36.063378 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:36.057896 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9" podUID="4f2b3150-1217-44bc-976a-65c86263223a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Oct 30 19:44:36.717938 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:36.717899 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-8nrtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 30 19:44:36.718343 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:36.717967 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8nrtp" podUID="b5978bc5-3ff8-4f4a-b85f-58b737179fee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 30 19:44:37.057787 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:37.057748 4292 patch_prober.go:29] interesting pod/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Oct 30 19:44:37.058102 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:37.058071 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9" podUID="4f2b3150-1217-44bc-976a-65c86263223a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Oct 30 19:44:37.717525 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:37.717487 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-8nrtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 30 19:44:37.717844 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:37.717802 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8nrtp" podUID="b5978bc5-3ff8-4f4a-b85f-58b737179fee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 30 19:44:38.057832 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:38.057790 4292 patch_prober.go:29] interesting pod/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Oct 30 19:44:38.058459 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:38.058426 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9" podUID="4f2b3150-1217-44bc-976a-65c86263223a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Oct 30 19:44:38.716775 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:38.716415 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-8nrtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 30 19:44:38.716775 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:38.716467 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8nrtp" podUID="b5978bc5-3ff8-4f4a-b85f-58b737179fee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 30 19:44:39.059216 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:39.058767 4292 patch_prober.go:29] interesting pod/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Oct 30 19:44:39.059216 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:39.058836 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9" podUID="4f2b3150-1217-44bc-976a-65c86263223a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Oct 30 19:44:39.716427 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:39.716388 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-8nrtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 30 19:44:39.717048 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:39.717012 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8nrtp" podUID="b5978bc5-3ff8-4f4a-b85f-58b737179fee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 30 19:44:40.076541 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:40.076502 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9" Oct 30 19:44:40.077043 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:40.076932 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9" Oct 30 19:44:40.077966 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:40.077944 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9" Oct 30 19:44:40.716772 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:40.716735 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-8nrtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 30 19:44:40.717116 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:40.717080 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8nrtp" podUID="b5978bc5-3ff8-4f4a-b85f-58b737179fee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 30 19:44:41.717231 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:41.717194 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-8nrtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 30 19:44:41.717747 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:41.717274 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8nrtp" podUID="b5978bc5-3ff8-4f4a-b85f-58b737179fee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 30 19:44:42.717185 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:42.717147 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-8nrtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 30 19:44:42.717441 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:42.717227 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8nrtp" podUID="b5978bc5-3ff8-4f4a-b85f-58b737179fee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 30 19:44:43.224499 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:44:43.224371 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:44:43.862210 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:43.862168 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8nrtp" Oct 30 19:44:43.862885 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:43.862862 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8nrtp" Oct 30 19:44:43.863651 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:43.863629 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8nrtp" Oct 30 19:44:52.392168 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:52.392126 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-bbfa07/distinct-w3-bbfa07-kserve-67bb47494b-t8t7l"] Oct 30 19:44:52.396036 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:52.396016 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bbfa07/distinct-w3-bbfa07-kserve-67bb47494b-t8t7l" Oct 30 19:44:52.492595 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:52.492562 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-bbfa07\"/\"distinct-w3-bbfa07-kserve-self-signed-certs\"" Oct 30 19:44:52.494916 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:52.494875 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-bbfa07/distinct-w3-bbfa07-kserve-67bb47494b-t8t7l"] Oct 30 19:44:52.518778 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:52.518740 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16def41a-a612-4a4c-ada9-bb170b066af2-model-cache\") pod \"distinct-w3-bbfa07-kserve-67bb47494b-t8t7l\" (UID: \"16def41a-a612-4a4c-ada9-bb170b066af2\") " pod="e2e-models-e2e-worker-w3-bbfa07/distinct-w3-bbfa07-kserve-67bb47494b-t8t7l" Oct 30 19:44:52.518947 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:52.518862 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2wlx\" (UniqueName: \"kubernetes.io/projected/16def41a-a612-4a4c-ada9-bb170b066af2-kube-api-access-w2wlx\") pod \"distinct-w3-bbfa07-kserve-67bb47494b-t8t7l\" (UID: \"16def41a-a612-4a4c-ada9-bb170b066af2\") " pod="e2e-models-e2e-worker-w3-bbfa07/distinct-w3-bbfa07-kserve-67bb47494b-t8t7l" Oct 30 19:44:52.518947 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:52.518895 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16def41a-a612-4a4c-ada9-bb170b066af2-dshm\") pod \"distinct-w3-bbfa07-kserve-67bb47494b-t8t7l\" (UID: \"16def41a-a612-4a4c-ada9-bb170b066af2\") " pod="e2e-models-e2e-worker-w3-bbfa07/distinct-w3-bbfa07-kserve-67bb47494b-t8t7l" Oct 30 19:44:52.518947 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:52.518919 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16def41a-a612-4a4c-ada9-bb170b066af2-home\") pod \"distinct-w3-bbfa07-kserve-67bb47494b-t8t7l\" (UID: \"16def41a-a612-4a4c-ada9-bb170b066af2\") " pod="e2e-models-e2e-worker-w3-bbfa07/distinct-w3-bbfa07-kserve-67bb47494b-t8t7l" Oct 30 19:44:52.519074 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:52.519019 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16def41a-a612-4a4c-ada9-bb170b066af2-tls-certs\") pod \"distinct-w3-bbfa07-kserve-67bb47494b-t8t7l\" (UID: \"16def41a-a612-4a4c-ada9-bb170b066af2\") " pod="e2e-models-e2e-worker-w3-bbfa07/distinct-w3-bbfa07-kserve-67bb47494b-t8t7l" Oct 30 19:44:52.519074 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:52.519053 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16def41a-a612-4a4c-ada9-bb170b066af2-tmp-dir\") pod \"distinct-w3-bbfa07-kserve-67bb47494b-t8t7l\" (UID: \"16def41a-a612-4a4c-ada9-bb170b066af2\") " pod="e2e-models-e2e-worker-w3-bbfa07/distinct-w3-bbfa07-kserve-67bb47494b-t8t7l" Oct 30 19:44:52.620144 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:52.620107 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16def41a-a612-4a4c-ada9-bb170b066af2-model-cache\") pod \"distinct-w3-bbfa07-kserve-67bb47494b-t8t7l\" (UID: \"16def41a-a612-4a4c-ada9-bb170b066af2\") " pod="e2e-models-e2e-worker-w3-bbfa07/distinct-w3-bbfa07-kserve-67bb47494b-t8t7l" Oct 30 19:44:52.620377 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:52.620176 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2wlx\" (UniqueName: \"kubernetes.io/projected/16def41a-a612-4a4c-ada9-bb170b066af2-kube-api-access-w2wlx\") pod \"distinct-w3-bbfa07-kserve-67bb47494b-t8t7l\" (UID: \"16def41a-a612-4a4c-ada9-bb170b066af2\") " pod="e2e-models-e2e-worker-w3-bbfa07/distinct-w3-bbfa07-kserve-67bb47494b-t8t7l" Oct 30 19:44:52.620377 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:52.620202 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16def41a-a612-4a4c-ada9-bb170b066af2-dshm\") pod \"distinct-w3-bbfa07-kserve-67bb47494b-t8t7l\" (UID: \"16def41a-a612-4a4c-ada9-bb170b066af2\") " pod="e2e-models-e2e-worker-w3-bbfa07/distinct-w3-bbfa07-kserve-67bb47494b-t8t7l" Oct 30 19:44:52.620377 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:52.620220 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16def41a-a612-4a4c-ada9-bb170b066af2-home\") pod \"distinct-w3-bbfa07-kserve-67bb47494b-t8t7l\" (UID: \"16def41a-a612-4a4c-ada9-bb170b066af2\") " pod="e2e-models-e2e-worker-w3-bbfa07/distinct-w3-bbfa07-kserve-67bb47494b-t8t7l" Oct 30 19:44:52.620377 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:52.620250 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16def41a-a612-4a4c-ada9-bb170b066af2-tls-certs\") pod \"distinct-w3-bbfa07-kserve-67bb47494b-t8t7l\" (UID: \"16def41a-a612-4a4c-ada9-bb170b066af2\") " pod="e2e-models-e2e-worker-w3-bbfa07/distinct-w3-bbfa07-kserve-67bb47494b-t8t7l" Oct 30 19:44:52.620377 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:52.620275 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16def41a-a612-4a4c-ada9-bb170b066af2-tmp-dir\") pod \"distinct-w3-bbfa07-kserve-67bb47494b-t8t7l\" (UID: \"16def41a-a612-4a4c-ada9-bb170b066af2\") " pod="e2e-models-e2e-worker-w3-bbfa07/distinct-w3-bbfa07-kserve-67bb47494b-t8t7l" Oct 30 19:44:52.620721 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:52.620700 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16def41a-a612-4a4c-ada9-bb170b066af2-tmp-dir\") pod \"distinct-w3-bbfa07-kserve-67bb47494b-t8t7l\" (UID: \"16def41a-a612-4a4c-ada9-bb170b066af2\") " pod="e2e-models-e2e-worker-w3-bbfa07/distinct-w3-bbfa07-kserve-67bb47494b-t8t7l" Oct 30 19:44:52.620813 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:52.620787 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16def41a-a612-4a4c-ada9-bb170b066af2-home\") pod \"distinct-w3-bbfa07-kserve-67bb47494b-t8t7l\" (UID: \"16def41a-a612-4a4c-ada9-bb170b066af2\") " pod="e2e-models-e2e-worker-w3-bbfa07/distinct-w3-bbfa07-kserve-67bb47494b-t8t7l" Oct 30 19:44:52.620988 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:52.620963 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16def41a-a612-4a4c-ada9-bb170b066af2-model-cache\") pod \"distinct-w3-bbfa07-kserve-67bb47494b-t8t7l\" (UID: \"16def41a-a612-4a4c-ada9-bb170b066af2\") " pod="e2e-models-e2e-worker-w3-bbfa07/distinct-w3-bbfa07-kserve-67bb47494b-t8t7l" Oct 30 19:44:52.623200 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:52.623178 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16def41a-a612-4a4c-ada9-bb170b066af2-dshm\") pod \"distinct-w3-bbfa07-kserve-67bb47494b-t8t7l\" (UID: \"16def41a-a612-4a4c-ada9-bb170b066af2\") " pod="e2e-models-e2e-worker-w3-bbfa07/distinct-w3-bbfa07-kserve-67bb47494b-t8t7l" Oct 30 19:44:52.623581 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:52.623559 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16def41a-a612-4a4c-ada9-bb170b066af2-tls-certs\") pod \"distinct-w3-bbfa07-kserve-67bb47494b-t8t7l\" (UID: \"16def41a-a612-4a4c-ada9-bb170b066af2\") " pod="e2e-models-e2e-worker-w3-bbfa07/distinct-w3-bbfa07-kserve-67bb47494b-t8t7l" Oct 30 19:44:52.678974 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:52.678885 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2wlx\" (UniqueName: \"kubernetes.io/projected/16def41a-a612-4a4c-ada9-bb170b066af2-kube-api-access-w2wlx\") pod \"distinct-w3-bbfa07-kserve-67bb47494b-t8t7l\" (UID: \"16def41a-a612-4a4c-ada9-bb170b066af2\") " pod="e2e-models-e2e-worker-w3-bbfa07/distinct-w3-bbfa07-kserve-67bb47494b-t8t7l" Oct 30 19:44:52.708457 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:52.708411 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bbfa07/distinct-w3-bbfa07-kserve-67bb47494b-t8t7l" Oct 30 19:44:53.102158 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:53.102128 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-bbfa07/distinct-w3-bbfa07-kserve-67bb47494b-t8t7l"] Oct 30 19:44:53.103272 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:44:53.103244 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod16def41a_a612_4a4c_ada9_bb170b066af2.slice/crio-70ae3e1d114c53f8b562bff44ca512184fbb822f1da354c456f5f24c4ce7c2b4 WatchSource:0}: Error finding container 70ae3e1d114c53f8b562bff44ca512184fbb822f1da354c456f5f24c4ce7c2b4: Status 404 returned error can't find the container with id 70ae3e1d114c53f8b562bff44ca512184fbb822f1da354c456f5f24c4ce7c2b4 Oct 30 19:44:53.584021 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:53.583979 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bbfa07/distinct-w3-bbfa07-kserve-67bb47494b-t8t7l" event={"ID":"16def41a-a612-4a4c-ada9-bb170b066af2","Type":"ContainerStarted","Data":"dcaf4828ece39711e3074a02f18ffcf0c67c200e3c00ddce8b15ee87b8652547"} Oct 30 19:44:53.584021 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:53.584025 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bbfa07/distinct-w3-bbfa07-kserve-67bb47494b-t8t7l" event={"ID":"16def41a-a612-4a4c-ada9-bb170b066af2","Type":"ContainerStarted","Data":"70ae3e1d114c53f8b562bff44ca512184fbb822f1da354c456f5f24c4ce7c2b4"} Oct 30 19:44:53.695667 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:53.695605 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-bbfa07/distinct-w3-bbfa07-kserve-67bb47494b-t8t7l" podStartSLOduration=1.695589768 podStartE2EDuration="1.695589768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:53.694734344 +0000 UTC m=+1197.213049802" watchObservedRunningTime="2026-10-30 19:44:53.695589768 +0000 UTC m=+1197.213905226" Oct 30 19:44:53.709192 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:44:53.709156 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bbfa07/distinct-w3-bbfa07-kserve-67bb47494b-t8t7l" Oct 30 19:45:00.185055 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:00.185012 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-98a19b-29889825-jgrdr"] Oct 30 19:45:00.191872 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:00.191844 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-98a19b-29889825-jgrdr" Oct 30 19:45:00.207053 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:00.207025 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889825-67xqm"] Oct 30 19:45:00.215964 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:00.213191 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889825-67xqm" Oct 30 19:45:00.227272 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:00.227240 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96692d03-7429-499f-a4be-9674b0c709bc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-98a19b-29889825-jgrdr\" (UID: \"96692d03-7429-499f-a4be-9674b0c709bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-98a19b-29889825-jgrdr" Oct 30 19:45:00.227424 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:00.227314 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sqdl\" (UniqueName: \"kubernetes.io/projected/756dede2-453c-44a3-afa7-b57ccaa0f044-kube-api-access-7sqdl\") pod \"maas-api-key-cleanup-e2e-worker-w4-4ec609-29889825-67xqm\" (UID: \"756dede2-453c-44a3-afa7-b57ccaa0f044\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889825-67xqm" Oct 30 19:45:00.227477 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:00.227444 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/756dede2-453c-44a3-afa7-b57ccaa0f044-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-4ec609-29889825-67xqm\" (UID: \"756dede2-453c-44a3-afa7-b57ccaa0f044\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889825-67xqm" Oct 30 19:45:00.227477 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:00.227473 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp96l\" (UniqueName: \"kubernetes.io/projected/96692d03-7429-499f-a4be-9674b0c709bc-kube-api-access-qp96l\") pod \"maas-api-key-cleanup-e2e-worker-w1-98a19b-29889825-jgrdr\" (UID: \"96692d03-7429-499f-a4be-9674b0c709bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-98a19b-29889825-jgrdr" Oct 30 19:45:00.243343 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:00.243315 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\"" Oct 30 19:45:00.292382 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:00.291720 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-98a19b-29889825-jgrdr"] Oct 30 19:45:00.324911 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:00.324874 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889825-67xqm"] Oct 30 19:45:00.328235 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:00.328197 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sqdl\" (UniqueName: \"kubernetes.io/projected/756dede2-453c-44a3-afa7-b57ccaa0f044-kube-api-access-7sqdl\") pod \"maas-api-key-cleanup-e2e-worker-w4-4ec609-29889825-67xqm\" (UID: \"756dede2-453c-44a3-afa7-b57ccaa0f044\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889825-67xqm" Oct 30 19:45:00.328437 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:00.328243 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/756dede2-453c-44a3-afa7-b57ccaa0f044-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-4ec609-29889825-67xqm\" (UID: \"756dede2-453c-44a3-afa7-b57ccaa0f044\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889825-67xqm" Oct 30 19:45:00.328437 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:00.328266 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp96l\" (UniqueName: \"kubernetes.io/projected/96692d03-7429-499f-a4be-9674b0c709bc-kube-api-access-qp96l\") pod \"maas-api-key-cleanup-e2e-worker-w1-98a19b-29889825-jgrdr\" (UID: \"96692d03-7429-499f-a4be-9674b0c709bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-98a19b-29889825-jgrdr" Oct 30 19:45:00.328437 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:00.328337 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96692d03-7429-499f-a4be-9674b0c709bc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-98a19b-29889825-jgrdr\" (UID: \"96692d03-7429-499f-a4be-9674b0c709bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-98a19b-29889825-jgrdr" Oct 30 19:45:00.329061 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:00.329037 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/756dede2-453c-44a3-afa7-b57ccaa0f044-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-4ec609-29889825-67xqm\" (UID: \"756dede2-453c-44a3-afa7-b57ccaa0f044\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889825-67xqm" Oct 30 19:45:00.329529 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:00.329502 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96692d03-7429-499f-a4be-9674b0c709bc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-98a19b-29889825-jgrdr\" (UID: \"96692d03-7429-499f-a4be-9674b0c709bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-98a19b-29889825-jgrdr" Oct 30 19:45:00.388496 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:00.388423 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sqdl\" (UniqueName: \"kubernetes.io/projected/756dede2-453c-44a3-afa7-b57ccaa0f044-kube-api-access-7sqdl\") pod \"maas-api-key-cleanup-e2e-worker-w4-4ec609-29889825-67xqm\" (UID: \"756dede2-453c-44a3-afa7-b57ccaa0f044\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889825-67xqm" Oct 30 19:45:00.389661 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:00.389607 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp96l\" (UniqueName: \"kubernetes.io/projected/96692d03-7429-499f-a4be-9674b0c709bc-kube-api-access-qp96l\") pod \"maas-api-key-cleanup-e2e-worker-w1-98a19b-29889825-jgrdr\" (UID: \"96692d03-7429-499f-a4be-9674b0c709bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-98a19b-29889825-jgrdr" Oct 30 19:45:00.519378 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:00.519315 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-98a19b-29889825-jgrdr" Oct 30 19:45:00.528864 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:00.528832 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889825-67xqm" Oct 30 19:45:00.939846 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:00.935583 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-98a19b-29889825-jgrdr"] Oct 30 19:45:00.976379 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:00.975784 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889825-67xqm"] Oct 30 19:45:00.978398 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:45:00.978362 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod756dede2_453c_44a3_afa7_b57ccaa0f044.slice/crio-46dabf43e62a3c39c4f17af039f8e05a28f863d41e6368d613a651e82b1f7233 WatchSource:0}: Error finding container 46dabf43e62a3c39c4f17af039f8e05a28f863d41e6368d613a651e82b1f7233: Status 404 returned error can't find the container with id 46dabf43e62a3c39c4f17af039f8e05a28f863d41e6368d613a651e82b1f7233 Oct 30 19:45:01.649077 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:01.649037 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889825-67xqm" event={"ID":"756dede2-453c-44a3-afa7-b57ccaa0f044","Type":"ContainerStarted","Data":"36ca6b920ab6c306db097265d16060584861e27b6a971309a3500a901bed03d0"} Oct 30 19:45:01.649626 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:01.649104 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889825-67xqm" event={"ID":"756dede2-453c-44a3-afa7-b57ccaa0f044","Type":"ContainerStarted","Data":"46dabf43e62a3c39c4f17af039f8e05a28f863d41e6368d613a651e82b1f7233"} Oct 30 19:45:01.652049 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:01.652016 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-98a19b-29889825-jgrdr" event={"ID":"96692d03-7429-499f-a4be-9674b0c709bc","Type":"ContainerStarted","Data":"afd034bdb552f43390501df2d7bc30ae15c91a817d95ad08096abffd459a5a63"} Oct 30 19:45:01.652170 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:01.652074 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-98a19b-29889825-jgrdr" event={"ID":"96692d03-7429-499f-a4be-9674b0c709bc","Type":"ContainerStarted","Data":"824da79f273e8b13b5966f61c2223a70f16ebaa9ac413e7ffaff131784482dd4"} Oct 30 19:45:01.763688 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:01.763632 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889825-67xqm" podStartSLOduration=1.7636072409999999 podStartE2EDuration="1.763607241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:01.762270594 +0000 UTC m=+1205.280586067" watchObservedRunningTime="2026-10-30 19:45:01.763607241 +0000 UTC m=+1205.281922698" Oct 30 19:45:01.880743 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:01.880692 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-98a19b-29889825-jgrdr" podStartSLOduration=1.880676857 podStartE2EDuration="1.880676857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:01.877719498 +0000 UTC m=+1205.396034956" watchObservedRunningTime="2026-10-30 19:45:01.880676857 +0000 UTC m=+1205.398992313" Oct 30 19:45:02.658860 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:02.658824 4292 generic.go:356] "Generic (PLEG): container finished" podID="756dede2-453c-44a3-afa7-b57ccaa0f044" containerID="36ca6b920ab6c306db097265d16060584861e27b6a971309a3500a901bed03d0" exitCode=60 Oct 30 19:45:02.659387 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:02.658920 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889825-67xqm" event={"ID":"756dede2-453c-44a3-afa7-b57ccaa0f044","Type":"ContainerDied","Data":"36ca6b920ab6c306db097265d16060584861e27b6a971309a3500a901bed03d0"} Oct 30 19:45:02.659387 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:02.659217 4292 scope.go:122] "RemoveContainer" containerID="36ca6b920ab6c306db097265d16060584861e27b6a971309a3500a901bed03d0" Oct 30 19:45:02.661748 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:02.661724 4292 generic.go:356] "Generic (PLEG): container finished" podID="96692d03-7429-499f-a4be-9674b0c709bc" containerID="afd034bdb552f43390501df2d7bc30ae15c91a817d95ad08096abffd459a5a63" exitCode=60 Oct 30 19:45:02.661873 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:02.661796 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-98a19b-29889825-jgrdr" event={"ID":"96692d03-7429-499f-a4be-9674b0c709bc","Type":"ContainerDied","Data":"afd034bdb552f43390501df2d7bc30ae15c91a817d95ad08096abffd459a5a63"} Oct 30 19:45:02.662038 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:02.662001 4292 scope.go:122] "RemoveContainer" containerID="afd034bdb552f43390501df2d7bc30ae15c91a817d95ad08096abffd459a5a63" Oct 30 19:45:02.709450 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:02.709403 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-bbfa07/distinct-w3-bbfa07-kserve-67bb47494b-t8t7l" Oct 30 19:45:02.722754 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:02.722717 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-bbfa07/distinct-w3-bbfa07-kserve-67bb47494b-t8t7l" Oct 30 19:45:03.669665 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:03.669630 4292 generic.go:356] "Generic (PLEG): container finished" podID="756dede2-453c-44a3-afa7-b57ccaa0f044" containerID="166d1cf6f1dd48661c897f84baaa3fe2f220d4883d03b9fc9e9789e27a458105" exitCode=60 Oct 30 19:45:03.670133 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:03.669714 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889825-67xqm" event={"ID":"756dede2-453c-44a3-afa7-b57ccaa0f044","Type":"ContainerDied","Data":"166d1cf6f1dd48661c897f84baaa3fe2f220d4883d03b9fc9e9789e27a458105"} Oct 30 19:45:03.670133 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:03.669770 4292 scope.go:122] "RemoveContainer" containerID="36ca6b920ab6c306db097265d16060584861e27b6a971309a3500a901bed03d0" Oct 30 19:45:03.670133 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:03.670083 4292 scope.go:122] "RemoveContainer" containerID="166d1cf6f1dd48661c897f84baaa3fe2f220d4883d03b9fc9e9789e27a458105" Oct 30 19:45:03.670410 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:45:03.670380 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-4ec609-29889825-67xqm_odh-ai-gateway-infra(756dede2-453c-44a3-afa7-b57ccaa0f044)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889825-67xqm" podUID="756dede2-453c-44a3-afa7-b57ccaa0f044" Oct 30 19:45:03.671633 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:03.671614 4292 generic.go:356] "Generic (PLEG): container finished" podID="96692d03-7429-499f-a4be-9674b0c709bc" containerID="aab0b790f5bca0df5d16a1e88a9b5047331b96d2dff7679d2e668400fe6159f2" exitCode=60 Oct 30 19:45:03.671710 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:03.671693 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-98a19b-29889825-jgrdr" event={"ID":"96692d03-7429-499f-a4be-9674b0c709bc","Type":"ContainerDied","Data":"aab0b790f5bca0df5d16a1e88a9b5047331b96d2dff7679d2e668400fe6159f2"} Oct 30 19:45:03.671930 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:03.671918 4292 scope.go:122] "RemoveContainer" containerID="aab0b790f5bca0df5d16a1e88a9b5047331b96d2dff7679d2e668400fe6159f2" Oct 30 19:45:03.672156 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:45:03.672117 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-98a19b-29889825-jgrdr_odh-ai-gateway-infra(96692d03-7429-499f-a4be-9674b0c709bc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-98a19b-29889825-jgrdr" podUID="96692d03-7429-499f-a4be-9674b0c709bc" Oct 30 19:45:03.681371 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:03.681336 4292 scope.go:122] "RemoveContainer" containerID="afd034bdb552f43390501df2d7bc30ae15c91a817d95ad08096abffd459a5a63" Oct 30 19:45:03.683233 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:03.683214 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bbfa07/distinct-w3-bbfa07-kserve-67bb47494b-t8t7l" Oct 30 19:45:04.679579 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:04.679549 4292 scope.go:122] "RemoveContainer" containerID="166d1cf6f1dd48661c897f84baaa3fe2f220d4883d03b9fc9e9789e27a458105" Oct 30 19:45:04.680019 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:45:04.679857 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-4ec609-29889825-67xqm_odh-ai-gateway-infra(756dede2-453c-44a3-afa7-b57ccaa0f044)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889825-67xqm" podUID="756dede2-453c-44a3-afa7-b57ccaa0f044" Oct 30 19:45:04.681169 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:04.681144 4292 scope.go:122] "RemoveContainer" containerID="aab0b790f5bca0df5d16a1e88a9b5047331b96d2dff7679d2e668400fe6159f2" Oct 30 19:45:04.681449 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:45:04.681425 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-98a19b-29889825-jgrdr_odh-ai-gateway-infra(96692d03-7429-499f-a4be-9674b0c709bc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-98a19b-29889825-jgrdr" podUID="96692d03-7429-499f-a4be-9674b0c709bc" Oct 30 19:45:13.219814 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:45:13.219725 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:45:14.582254 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:14.582218 4292 scope.go:122] "RemoveContainer" containerID="aab0b790f5bca0df5d16a1e88a9b5047331b96d2dff7679d2e668400fe6159f2" Oct 30 19:45:15.745242 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:15.745156 4292 generic.go:356] "Generic (PLEG): container finished" podID="96692d03-7429-499f-a4be-9674b0c709bc" containerID="b09f06796f6bf3a7718d1bf25b44fcab2c6090b5eb52a0d4016306d48b26802e" exitCode=60 Oct 30 19:45:15.745242 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:15.745226 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-98a19b-29889825-jgrdr" event={"ID":"96692d03-7429-499f-a4be-9674b0c709bc","Type":"ContainerDied","Data":"b09f06796f6bf3a7718d1bf25b44fcab2c6090b5eb52a0d4016306d48b26802e"} Oct 30 19:45:15.745737 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:15.745300 4292 scope.go:122] "RemoveContainer" containerID="aab0b790f5bca0df5d16a1e88a9b5047331b96d2dff7679d2e668400fe6159f2" Oct 30 19:45:15.745737 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:15.745695 4292 scope.go:122] "RemoveContainer" containerID="b09f06796f6bf3a7718d1bf25b44fcab2c6090b5eb52a0d4016306d48b26802e" Oct 30 19:45:15.746011 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:45:15.745990 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-98a19b-29889825-jgrdr_odh-ai-gateway-infra(96692d03-7429-499f-a4be-9674b0c709bc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-98a19b-29889825-jgrdr" podUID="96692d03-7429-499f-a4be-9674b0c709bc" Oct 30 19:45:16.728080 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:45:16.728046 4292 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-294332af605e: secret "agc-cred-294332af605e" not found Oct 30 19:45:16.728312 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:45:16.728285 4292 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-788b6c6b4d-59ggs: secret "agc-cred-294332af605e" not found Oct 30 19:45:16.728421 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:45:16.728388 4292 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a179efae-1f42-4b73-b2d7-70f1474c3c9f-provider-credentials podName:a179efae-1f42-4b73-b2d7-70f1474c3c9f nodeName:}" failed. No retries permitted until 2026-10-30 19:45:17.228365754 +0000 UTC m=+1220.746681208 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a179efae-1f42-4b73-b2d7-70f1474c3c9f-provider-credentials") pod "payload-processing-external-model-788b6c6b4d-59ggs" (UID: "a179efae-1f42-4b73-b2d7-70f1474c3c9f") : secret "agc-cred-294332af605e" not found Oct 30 19:45:16.888239 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:16.888193 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-98a19b-29889825-jgrdr"] Oct 30 19:45:17.021581 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:17.021543 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-98a19b-29889825-jgrdr" Oct 30 19:45:17.132185 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:17.132133 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96692d03-7429-499f-a4be-9674b0c709bc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96692d03-7429-499f-a4be-9674b0c709bc-maas-api-service-ca\") pod \"96692d03-7429-499f-a4be-9674b0c709bc\" (UID: \"96692d03-7429-499f-a4be-9674b0c709bc\") " Oct 30 19:45:17.132387 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:17.132292 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96692d03-7429-499f-a4be-9674b0c709bc-kube-api-access-qp96l\" (UniqueName: \"kubernetes.io/projected/96692d03-7429-499f-a4be-9674b0c709bc-kube-api-access-qp96l\") pod \"96692d03-7429-499f-a4be-9674b0c709bc\" (UID: \"96692d03-7429-499f-a4be-9674b0c709bc\") " Oct 30 19:45:17.132559 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:17.132533 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96692d03-7429-499f-a4be-9674b0c709bc-maas-api-service-ca" pod "96692d03-7429-499f-a4be-9674b0c709bc" (UID: "96692d03-7429-499f-a4be-9674b0c709bc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:17.132785 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:17.132762 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96692d03-7429-499f-a4be-9674b0c709bc-maas-api-service-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:17.134466 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:17.134440 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96692d03-7429-499f-a4be-9674b0c709bc-kube-api-access-qp96l" pod "96692d03-7429-499f-a4be-9674b0c709bc" (UID: "96692d03-7429-499f-a4be-9674b0c709bc"). InnerVolumeSpecName "kube-api-access-qp96l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:17.233587 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:17.233551 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp96l\" (UniqueName: \"kubernetes.io/projected/96692d03-7429-499f-a4be-9674b0c709bc-kube-api-access-qp96l\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:17.233587 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:45:17.233568 4292 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-294332af605e: secret "agc-cred-294332af605e" not found Oct 30 19:45:17.233587 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:45:17.233590 4292 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-788b6c6b4d-59ggs: secret "agc-cred-294332af605e" not found Oct 30 19:45:17.233826 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:45:17.233645 4292 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a179efae-1f42-4b73-b2d7-70f1474c3c9f-provider-credentials podName:a179efae-1f42-4b73-b2d7-70f1474c3c9f nodeName:}" failed. No retries permitted until 2026-10-30 19:45:18.233630725 +0000 UTC m=+1221.751946182 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a179efae-1f42-4b73-b2d7-70f1474c3c9f-provider-credentials") pod "payload-processing-external-model-788b6c6b4d-59ggs" (UID: "a179efae-1f42-4b73-b2d7-70f1474c3c9f") : secret "agc-cred-294332af605e" not found Oct 30 19:45:17.760447 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:17.760411 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-98a19b-29889825-jgrdr" event={"ID":"96692d03-7429-499f-a4be-9674b0c709bc","Type":"ContainerDied","Data":"824da79f273e8b13b5966f61c2223a70f16ebaa9ac413e7ffaff131784482dd4"} Oct 30 19:45:17.760639 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:17.760476 4292 scope.go:122] "RemoveContainer" containerID="b09f06796f6bf3a7718d1bf25b44fcab2c6090b5eb52a0d4016306d48b26802e" Oct 30 19:45:17.760639 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:17.760478 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-98a19b-29889825-jgrdr" Oct 30 19:45:17.920122 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:17.920086 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-98a19b-29889825-jgrdr"] Oct 30 19:45:17.967336 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:17.967296 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-98a19b-29889825-jgrdr"] Oct 30 19:45:17.969421 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:17.969397 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4"] Oct 30 19:45:17.969884 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:17.969868 4292 cpu_manager.go:414] "containerMap: removing container" podUID="96692d03-7429-499f-a4be-9674b0c709bc" containerName="cleanup" Oct 30 19:45:17.969932 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:17.969886 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96692d03-7429-499f-a4be-9674b0c709bc" containerName="cleanup" Oct 30 19:45:17.969932 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:17.969900 4292 cpu_manager.go:414] "containerMap: removing container" podUID="96692d03-7429-499f-a4be-9674b0c709bc" containerName="cleanup" Oct 30 19:45:17.969932 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:17.969906 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96692d03-7429-499f-a4be-9674b0c709bc" containerName="cleanup" Oct 30 19:45:17.970022 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:17.969994 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="96692d03-7429-499f-a4be-9674b0c709bc" containerName="cleanup" Oct 30 19:45:17.970022 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:17.970006 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="96692d03-7429-499f-a4be-9674b0c709bc" containerName="cleanup" Oct 30 19:45:17.974516 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:17.974497 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4" Oct 30 19:45:18.034717 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:18.034683 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4"] Oct 30 19:45:18.042821 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:18.042776 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5cde15b-cd07-4ce9-8b96-95673db304a8-istio-token\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4\" (UID: \"e5cde15b-cd07-4ce9-8b96-95673db304a8\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4" Oct 30 19:45:18.042980 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:18.042836 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5cde15b-cd07-4ce9-8b96-95673db304a8-istio-podinfo\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4\" (UID: \"e5cde15b-cd07-4ce9-8b96-95673db304a8\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4" Oct 30 19:45:18.042980 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:18.042913 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b78qj\" (UniqueName: \"kubernetes.io/projected/e5cde15b-cd07-4ce9-8b96-95673db304a8-kube-api-access-b78qj\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4\" (UID: \"e5cde15b-cd07-4ce9-8b96-95673db304a8\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4" Oct 30 19:45:18.043054 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:18.042994 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cde15b-cd07-4ce9-8b96-95673db304a8-credential-socket\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4\" (UID: \"e5cde15b-cd07-4ce9-8b96-95673db304a8\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4" Oct 30 19:45:18.043054 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:18.043034 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cde15b-cd07-4ce9-8b96-95673db304a8-workload-socket\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4\" (UID: \"e5cde15b-cd07-4ce9-8b96-95673db304a8\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4" Oct 30 19:45:18.043122 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:18.043101 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5cde15b-cd07-4ce9-8b96-95673db304a8-istio-data\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4\" (UID: \"e5cde15b-cd07-4ce9-8b96-95673db304a8\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4" Oct 30 19:45:18.043162 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:18.043133 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5cde15b-cd07-4ce9-8b96-95673db304a8-istio-envoy\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4\" (UID: \"e5cde15b-cd07-4ce9-8b96-95673db304a8\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4" Oct 30 19:45:18.043305 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:18.043286 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5cde15b-cd07-4ce9-8b96-95673db304a8-istiod-ca-cert\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4\" (UID: \"e5cde15b-cd07-4ce9-8b96-95673db304a8\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4" Oct 30 19:45:18.043397 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:18.043364 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5cde15b-cd07-4ce9-8b96-95673db304a8-workload-certs\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4\" (UID: \"e5cde15b-cd07-4ce9-8b96-95673db304a8\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4" Oct 30 19:45:18.144376 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:18.144315 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5cde15b-cd07-4ce9-8b96-95673db304a8-istiod-ca-cert\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4\" (UID: \"e5cde15b-cd07-4ce9-8b96-95673db304a8\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4" Oct 30 19:45:18.144590 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:18.144405 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5cde15b-cd07-4ce9-8b96-95673db304a8-workload-certs\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4\" (UID: \"e5cde15b-cd07-4ce9-8b96-95673db304a8\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4" Oct 30 19:45:18.144590 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:18.144452 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5cde15b-cd07-4ce9-8b96-95673db304a8-istio-token\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4\" (UID: \"e5cde15b-cd07-4ce9-8b96-95673db304a8\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4" Oct 30 19:45:18.144590 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:18.144472 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5cde15b-cd07-4ce9-8b96-95673db304a8-istio-podinfo\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4\" (UID: \"e5cde15b-cd07-4ce9-8b96-95673db304a8\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4" Oct 30 19:45:18.144590 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:18.144490 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b78qj\" (UniqueName: \"kubernetes.io/projected/e5cde15b-cd07-4ce9-8b96-95673db304a8-kube-api-access-b78qj\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4\" (UID: \"e5cde15b-cd07-4ce9-8b96-95673db304a8\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4" Oct 30 19:45:18.144590 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:18.144516 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cde15b-cd07-4ce9-8b96-95673db304a8-credential-socket\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4\" (UID: \"e5cde15b-cd07-4ce9-8b96-95673db304a8\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4" Oct 30 19:45:18.144590 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:18.144549 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cde15b-cd07-4ce9-8b96-95673db304a8-workload-socket\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4\" (UID: \"e5cde15b-cd07-4ce9-8b96-95673db304a8\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4" Oct 30 19:45:18.144922 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:18.144603 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5cde15b-cd07-4ce9-8b96-95673db304a8-istio-data\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4\" (UID: \"e5cde15b-cd07-4ce9-8b96-95673db304a8\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4" Oct 30 19:45:18.144922 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:18.144644 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5cde15b-cd07-4ce9-8b96-95673db304a8-istio-envoy\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4\" (UID: \"e5cde15b-cd07-4ce9-8b96-95673db304a8\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4" Oct 30 19:45:18.145040 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:18.145011 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cde15b-cd07-4ce9-8b96-95673db304a8-workload-socket\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4\" (UID: \"e5cde15b-cd07-4ce9-8b96-95673db304a8\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4" Oct 30 19:45:18.145040 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:18.145024 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cde15b-cd07-4ce9-8b96-95673db304a8-credential-socket\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4\" (UID: \"e5cde15b-cd07-4ce9-8b96-95673db304a8\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4" Oct 30 19:45:18.145150 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:18.145085 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5cde15b-cd07-4ce9-8b96-95673db304a8-istio-data\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4\" (UID: \"e5cde15b-cd07-4ce9-8b96-95673db304a8\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4" Oct 30 19:45:18.145205 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:18.145157 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5cde15b-cd07-4ce9-8b96-95673db304a8-istiod-ca-cert\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4\" (UID: \"e5cde15b-cd07-4ce9-8b96-95673db304a8\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4" Oct 30 19:45:18.145205 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:18.145163 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5cde15b-cd07-4ce9-8b96-95673db304a8-workload-certs\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4\" (UID: \"e5cde15b-cd07-4ce9-8b96-95673db304a8\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4" Oct 30 19:45:18.146881 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:18.146847 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5cde15b-cd07-4ce9-8b96-95673db304a8-istio-envoy\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4\" (UID: \"e5cde15b-cd07-4ce9-8b96-95673db304a8\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4" Oct 30 19:45:18.147055 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:18.147032 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5cde15b-cd07-4ce9-8b96-95673db304a8-istio-podinfo\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4\" (UID: \"e5cde15b-cd07-4ce9-8b96-95673db304a8\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4" Oct 30 19:45:18.209265 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:18.209229 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5cde15b-cd07-4ce9-8b96-95673db304a8-istio-token\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4\" (UID: \"e5cde15b-cd07-4ce9-8b96-95673db304a8\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4" Oct 30 19:45:18.209763 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:18.209743 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b78qj\" (UniqueName: \"kubernetes.io/projected/e5cde15b-cd07-4ce9-8b96-95673db304a8-kube-api-access-b78qj\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4\" (UID: \"e5cde15b-cd07-4ce9-8b96-95673db304a8\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4" Oct 30 19:45:18.245680 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:45:18.245646 4292 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-294332af605e: secret "agc-cred-294332af605e" not found Oct 30 19:45:18.245680 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:45:18.245677 4292 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-788b6c6b4d-59ggs: secret "agc-cred-294332af605e" not found Oct 30 19:45:18.245890 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:45:18.245731 4292 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a179efae-1f42-4b73-b2d7-70f1474c3c9f-provider-credentials podName:a179efae-1f42-4b73-b2d7-70f1474c3c9f nodeName:}" failed. No retries permitted until 2026-10-30 19:45:20.245716387 +0000 UTC m=+1223.764031848 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a179efae-1f42-4b73-b2d7-70f1474c3c9f-provider-credentials") pod "payload-processing-external-model-788b6c6b4d-59ggs" (UID: "a179efae-1f42-4b73-b2d7-70f1474c3c9f") : secret "agc-cred-294332af605e" not found Oct 30 19:45:18.288777 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:18.288664 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4" Oct 30 19:45:18.576027 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:18.575938 4292 scope.go:122] "RemoveContainer" containerID="166d1cf6f1dd48661c897f84baaa3fe2f220d4883d03b9fc9e9789e27a458105" Oct 30 19:45:18.582403 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:18.582370 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96692d03-7429-499f-a4be-9674b0c709bc" path="/var/lib/kubelet/pods/96692d03-7429-499f-a4be-9674b0c709bc/volumes" Oct 30 19:45:18.718435 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:45:18.718395 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5cde15b_cd07_4ce9_8b96_95673db304a8.slice/crio-4b0d7da8ffcd2fce51fbf0c1ecdb05de69026f9b4964b8770ac1df7c60d40992 WatchSource:0}: Error finding container 4b0d7da8ffcd2fce51fbf0c1ecdb05de69026f9b4964b8770ac1df7c60d40992: Status 404 returned error can't find the container with id 4b0d7da8ffcd2fce51fbf0c1ecdb05de69026f9b4964b8770ac1df7c60d40992 Oct 30 19:45:18.719109 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:18.719084 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4"] Oct 30 19:45:18.720912 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:18.720882 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:45:18.720989 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:18.720949 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:45:18.720989 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:18.720979 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:45:18.768522 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:18.768491 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4" event={"ID":"e5cde15b-cd07-4ce9-8b96-95673db304a8","Type":"ContainerStarted","Data":"4b0d7da8ffcd2fce51fbf0c1ecdb05de69026f9b4964b8770ac1df7c60d40992"} Oct 30 19:45:19.793450 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:19.793414 4292 generic.go:356] "Generic (PLEG): container finished" podID="756dede2-453c-44a3-afa7-b57ccaa0f044" containerID="a9b5e7148bcaddf4a247d81053459cda47536e676b104ad5392267c87101cf2d" exitCode=60 Oct 30 19:45:19.793948 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:19.793527 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889825-67xqm" event={"ID":"756dede2-453c-44a3-afa7-b57ccaa0f044","Type":"ContainerDied","Data":"a9b5e7148bcaddf4a247d81053459cda47536e676b104ad5392267c87101cf2d"} Oct 30 19:45:19.793948 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:19.793572 4292 scope.go:122] "RemoveContainer" containerID="166d1cf6f1dd48661c897f84baaa3fe2f220d4883d03b9fc9e9789e27a458105" Oct 30 19:45:19.793948 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:19.793906 4292 scope.go:122] "RemoveContainer" containerID="a9b5e7148bcaddf4a247d81053459cda47536e676b104ad5392267c87101cf2d" Oct 30 19:45:19.794410 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:45:19.794156 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-4ec609-29889825-67xqm_odh-ai-gateway-infra(756dede2-453c-44a3-afa7-b57ccaa0f044)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889825-67xqm" podUID="756dede2-453c-44a3-afa7-b57ccaa0f044" Oct 30 19:45:19.796681 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:19.796650 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4" event={"ID":"e5cde15b-cd07-4ce9-8b96-95673db304a8","Type":"ContainerStarted","Data":"afb8e6622f70196ec501d99e73c066e4495f534a9b03d5ddbc205fcc8cc40110"} Oct 30 19:45:20.033628 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:20.033573 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4" podStartSLOduration=3.033558338 podStartE2EDuration="3.033558338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:20.031050871 +0000 UTC m=+1223.549366328" watchObservedRunningTime="2026-10-30 19:45:20.033558338 +0000 UTC m=+1223.551873795" Oct 30 19:45:20.289474 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:20.289384 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4" Oct 30 19:45:20.290007 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:45:20.289984 4292 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-294332af605e: secret "agc-cred-294332af605e" not found Oct 30 19:45:20.290081 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:45:20.290012 4292 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-788b6c6b4d-59ggs: secret "agc-cred-294332af605e" not found Oct 30 19:45:20.290081 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:45:20.290066 4292 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a179efae-1f42-4b73-b2d7-70f1474c3c9f-provider-credentials podName:a179efae-1f42-4b73-b2d7-70f1474c3c9f nodeName:}" failed. No retries permitted until 2026-10-30 19:45:24.290051227 +0000 UTC m=+1227.808366662 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a179efae-1f42-4b73-b2d7-70f1474c3c9f-provider-credentials") pod "payload-processing-external-model-788b6c6b4d-59ggs" (UID: "a179efae-1f42-4b73-b2d7-70f1474c3c9f") : secret "agc-cred-294332af605e" not found Oct 30 19:45:20.291084 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:20.291056 4292 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 19:45:20.291185 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:20.291108 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4" podUID="e5cde15b-cd07-4ce9-8b96-95673db304a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 19:45:20.914801 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:20.914766 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889825-67xqm"] Oct 30 19:45:21.085538 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:21.085510 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889825-67xqm" Oct 30 19:45:21.199977 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:21.199894 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/756dede2-453c-44a3-afa7-b57ccaa0f044-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/756dede2-453c-44a3-afa7-b57ccaa0f044-maas-api-service-ca\") pod \"756dede2-453c-44a3-afa7-b57ccaa0f044\" (UID: \"756dede2-453c-44a3-afa7-b57ccaa0f044\") " Oct 30 19:45:21.200579 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:21.200549 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/756dede2-453c-44a3-afa7-b57ccaa0f044-maas-api-service-ca" pod "756dede2-453c-44a3-afa7-b57ccaa0f044" (UID: "756dede2-453c-44a3-afa7-b57ccaa0f044"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:21.200744 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:21.200702 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/756dede2-453c-44a3-afa7-b57ccaa0f044-kube-api-access-7sqdl\" (UniqueName: \"kubernetes.io/projected/756dede2-453c-44a3-afa7-b57ccaa0f044-kube-api-access-7sqdl\") pod \"756dede2-453c-44a3-afa7-b57ccaa0f044\" (UID: \"756dede2-453c-44a3-afa7-b57ccaa0f044\") " Oct 30 19:45:21.201271 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:21.201210 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/756dede2-453c-44a3-afa7-b57ccaa0f044-maas-api-service-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:21.203428 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:21.203400 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/756dede2-453c-44a3-afa7-b57ccaa0f044-kube-api-access-7sqdl" pod "756dede2-453c-44a3-afa7-b57ccaa0f044" (UID: "756dede2-453c-44a3-afa7-b57ccaa0f044"). InnerVolumeSpecName "kube-api-access-7sqdl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:21.289331 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:21.289301 4292 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 19:45:21.289554 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:21.289460 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4" podUID="e5cde15b-cd07-4ce9-8b96-95673db304a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 19:45:21.302965 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:21.302934 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sqdl\" (UniqueName: \"kubernetes.io/projected/756dede2-453c-44a3-afa7-b57ccaa0f044-kube-api-access-7sqdl\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:21.811102 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:21.811069 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889825-67xqm" event={"ID":"756dede2-453c-44a3-afa7-b57ccaa0f044","Type":"ContainerDied","Data":"46dabf43e62a3c39c4f17af039f8e05a28f863d41e6368d613a651e82b1f7233"} Oct 30 19:45:21.811315 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:21.811116 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889825-67xqm" Oct 30 19:45:21.811315 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:21.811136 4292 scope.go:122] "RemoveContainer" containerID="a9b5e7148bcaddf4a247d81053459cda47536e676b104ad5392267c87101cf2d" Oct 30 19:45:21.977519 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:21.977472 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889825-67xqm"] Oct 30 19:45:21.979551 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:21.979529 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889825-67xqm"] Oct 30 19:45:22.288994 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:22.288957 4292 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 19:45:22.289190 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:22.289014 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4" podUID="e5cde15b-cd07-4ce9-8b96-95673db304a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 19:45:22.582798 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:22.582716 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="756dede2-453c-44a3-afa7-b57ccaa0f044" path="/var/lib/kubelet/pods/756dede2-453c-44a3-afa7-b57ccaa0f044/volumes" Oct 30 19:45:23.289340 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:23.289299 4292 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 19:45:23.289782 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:23.289380 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4" podUID="e5cde15b-cd07-4ce9-8b96-95673db304a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 19:45:24.289485 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:24.289446 4292 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 19:45:24.290054 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:24.289506 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4" podUID="e5cde15b-cd07-4ce9-8b96-95673db304a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 19:45:24.339536 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:45:24.339497 4292 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-449cc5558c4a: secret "agc-cred-449cc5558c4a" not found Oct 30 19:45:24.339536 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:45:24.339528 4292 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-788b6c6b4d-59ggs: secret "agc-cred-449cc5558c4a" not found Oct 30 19:45:24.339751 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:45:24.339589 4292 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a179efae-1f42-4b73-b2d7-70f1474c3c9f-provider-credentials podName:a179efae-1f42-4b73-b2d7-70f1474c3c9f nodeName:}" failed. No retries permitted until 2026-10-30 19:45:32.339569005 +0000 UTC m=+1235.857884444 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a179efae-1f42-4b73-b2d7-70f1474c3c9f-provider-credentials") pod "payload-processing-external-model-788b6c6b4d-59ggs" (UID: "a179efae-1f42-4b73-b2d7-70f1474c3c9f") : secret "agc-cred-449cc5558c4a" not found Oct 30 19:45:25.289101 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:25.289056 4292 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 19:45:25.289297 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:25.289171 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4" podUID="e5cde15b-cd07-4ce9-8b96-95673db304a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 19:45:26.293285 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:26.293255 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4" Oct 30 19:45:26.293712 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:26.293640 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4" Oct 30 19:45:26.294406 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:26.294389 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4" Oct 30 19:45:32.433030 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:45:32.432996 4292 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-449cc5558c4a: secret "agc-cred-449cc5558c4a" not found Oct 30 19:45:32.433030 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:45:32.433021 4292 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-788b6c6b4d-59ggs: secret "agc-cred-449cc5558c4a" not found Oct 30 19:45:32.433484 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:45:32.433080 4292 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a179efae-1f42-4b73-b2d7-70f1474c3c9f-provider-credentials podName:a179efae-1f42-4b73-b2d7-70f1474c3c9f nodeName:}" failed. No retries permitted until 2026-10-30 19:45:48.433065026 +0000 UTC m=+1251.951380461 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a179efae-1f42-4b73-b2d7-70f1474c3c9f-provider-credentials") pod "payload-processing-external-model-788b6c6b4d-59ggs" (UID: "a179efae-1f42-4b73-b2d7-70f1474c3c9f") : secret "agc-cred-449cc5558c4a" not found Oct 30 19:45:43.221477 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:45:43.221440 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:45:47.384660 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:47.384619 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s"] Oct 30 19:45:47.385090 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:47.385064 4292 cpu_manager.go:414] "containerMap: removing container" podUID="756dede2-453c-44a3-afa7-b57ccaa0f044" containerName="cleanup" Oct 30 19:45:47.385090 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:47.385076 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="756dede2-453c-44a3-afa7-b57ccaa0f044" containerName="cleanup" Oct 30 19:45:47.385090 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:47.385088 4292 cpu_manager.go:414] "containerMap: removing container" podUID="96692d03-7429-499f-a4be-9674b0c709bc" containerName="cleanup" Oct 30 19:45:47.385194 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:47.385094 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96692d03-7429-499f-a4be-9674b0c709bc" containerName="cleanup" Oct 30 19:45:47.385194 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:47.385117 4292 cpu_manager.go:414] "containerMap: removing container" podUID="756dede2-453c-44a3-afa7-b57ccaa0f044" containerName="cleanup" Oct 30 19:45:47.385194 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:47.385122 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="756dede2-453c-44a3-afa7-b57ccaa0f044" containerName="cleanup" Oct 30 19:45:47.385194 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:47.385188 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="756dede2-453c-44a3-afa7-b57ccaa0f044" containerName="cleanup" Oct 30 19:45:47.385194 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:47.385195 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="756dede2-453c-44a3-afa7-b57ccaa0f044" containerName="cleanup" Oct 30 19:45:47.385379 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:47.385205 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="96692d03-7429-499f-a4be-9674b0c709bc" containerName="cleanup" Oct 30 19:45:47.388345 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:47.388329 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s" Oct 30 19:45:47.433972 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:47.433938 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd0da58c-2f5a-4733-bea2-d2599a9931f2-credential-socket\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s\" (UID: \"dd0da58c-2f5a-4733-bea2-d2599a9931f2\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s" Oct 30 19:45:47.434179 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:47.433987 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd0da58c-2f5a-4733-bea2-d2599a9931f2-istio-podinfo\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s\" (UID: \"dd0da58c-2f5a-4733-bea2-d2599a9931f2\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s" Oct 30 19:45:47.434179 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:47.434048 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd0da58c-2f5a-4733-bea2-d2599a9931f2-istiod-ca-cert\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s\" (UID: \"dd0da58c-2f5a-4733-bea2-d2599a9931f2\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s" Oct 30 19:45:47.434179 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:47.434126 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd0da58c-2f5a-4733-bea2-d2599a9931f2-istio-envoy\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s\" (UID: \"dd0da58c-2f5a-4733-bea2-d2599a9931f2\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s" Oct 30 19:45:47.434362 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:47.434239 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd0da58c-2f5a-4733-bea2-d2599a9931f2-workload-socket\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s\" (UID: \"dd0da58c-2f5a-4733-bea2-d2599a9931f2\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s" Oct 30 19:45:47.434362 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:47.434268 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd0da58c-2f5a-4733-bea2-d2599a9931f2-istio-token\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s\" (UID: \"dd0da58c-2f5a-4733-bea2-d2599a9931f2\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s" Oct 30 19:45:47.434362 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:47.434313 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd0da58c-2f5a-4733-bea2-d2599a9931f2-istio-data\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s\" (UID: \"dd0da58c-2f5a-4733-bea2-d2599a9931f2\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s" Oct 30 19:45:47.434467 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:47.434392 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzbfl\" (UniqueName: \"kubernetes.io/projected/dd0da58c-2f5a-4733-bea2-d2599a9931f2-kube-api-access-gzbfl\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s\" (UID: \"dd0da58c-2f5a-4733-bea2-d2599a9931f2\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s" Oct 30 19:45:47.434467 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:47.434444 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd0da58c-2f5a-4733-bea2-d2599a9931f2-workload-certs\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s\" (UID: \"dd0da58c-2f5a-4733-bea2-d2599a9931f2\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s" Oct 30 19:45:47.504580 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:47.504535 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s"] Oct 30 19:45:47.535466 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:47.535425 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd0da58c-2f5a-4733-bea2-d2599a9931f2-istio-envoy\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s\" (UID: \"dd0da58c-2f5a-4733-bea2-d2599a9931f2\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s" Oct 30 19:45:47.535666 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:47.535552 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd0da58c-2f5a-4733-bea2-d2599a9931f2-workload-socket\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s\" (UID: \"dd0da58c-2f5a-4733-bea2-d2599a9931f2\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s" Oct 30 19:45:47.535666 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:47.535581 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd0da58c-2f5a-4733-bea2-d2599a9931f2-istio-token\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s\" (UID: \"dd0da58c-2f5a-4733-bea2-d2599a9931f2\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s" Oct 30 19:45:47.535666 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:47.535629 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd0da58c-2f5a-4733-bea2-d2599a9931f2-istio-data\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s\" (UID: \"dd0da58c-2f5a-4733-bea2-d2599a9931f2\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s" Oct 30 19:45:47.535837 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:47.535667 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzbfl\" (UniqueName: \"kubernetes.io/projected/dd0da58c-2f5a-4733-bea2-d2599a9931f2-kube-api-access-gzbfl\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s\" (UID: \"dd0da58c-2f5a-4733-bea2-d2599a9931f2\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s" Oct 30 19:45:47.535837 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:47.535726 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd0da58c-2f5a-4733-bea2-d2599a9931f2-workload-certs\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s\" (UID: \"dd0da58c-2f5a-4733-bea2-d2599a9931f2\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s" Oct 30 19:45:47.535837 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:47.535829 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd0da58c-2f5a-4733-bea2-d2599a9931f2-credential-socket\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s\" (UID: \"dd0da58c-2f5a-4733-bea2-d2599a9931f2\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s" Oct 30 19:45:47.536005 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:47.535868 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd0da58c-2f5a-4733-bea2-d2599a9931f2-istio-podinfo\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s\" (UID: \"dd0da58c-2f5a-4733-bea2-d2599a9931f2\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s" Oct 30 19:45:47.536005 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:47.535892 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd0da58c-2f5a-4733-bea2-d2599a9931f2-istiod-ca-cert\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s\" (UID: \"dd0da58c-2f5a-4733-bea2-d2599a9931f2\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s" Oct 30 19:45:47.536120 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:47.536069 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd0da58c-2f5a-4733-bea2-d2599a9931f2-workload-socket\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s\" (UID: \"dd0da58c-2f5a-4733-bea2-d2599a9931f2\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s" Oct 30 19:45:47.536120 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:47.536099 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd0da58c-2f5a-4733-bea2-d2599a9931f2-istio-data\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s\" (UID: \"dd0da58c-2f5a-4733-bea2-d2599a9931f2\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s" Oct 30 19:45:47.536530 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:47.536495 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd0da58c-2f5a-4733-bea2-d2599a9931f2-credential-socket\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s\" (UID: \"dd0da58c-2f5a-4733-bea2-d2599a9931f2\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s" Oct 30 19:45:47.536642 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:47.536530 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd0da58c-2f5a-4733-bea2-d2599a9931f2-workload-certs\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s\" (UID: \"dd0da58c-2f5a-4733-bea2-d2599a9931f2\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s" Oct 30 19:45:47.536832 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:47.536808 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd0da58c-2f5a-4733-bea2-d2599a9931f2-istiod-ca-cert\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s\" (UID: \"dd0da58c-2f5a-4733-bea2-d2599a9931f2\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s" Oct 30 19:45:47.538489 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:47.538469 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd0da58c-2f5a-4733-bea2-d2599a9931f2-istio-envoy\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s\" (UID: \"dd0da58c-2f5a-4733-bea2-d2599a9931f2\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s" Oct 30 19:45:47.538611 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:47.538503 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd0da58c-2f5a-4733-bea2-d2599a9931f2-istio-podinfo\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s\" (UID: \"dd0da58c-2f5a-4733-bea2-d2599a9931f2\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s" Oct 30 19:45:47.636439 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:47.636344 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd0da58c-2f5a-4733-bea2-d2599a9931f2-istio-token\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s\" (UID: \"dd0da58c-2f5a-4733-bea2-d2599a9931f2\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s" Oct 30 19:45:47.636645 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:47.636623 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzbfl\" (UniqueName: \"kubernetes.io/projected/dd0da58c-2f5a-4733-bea2-d2599a9931f2-kube-api-access-gzbfl\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s\" (UID: \"dd0da58c-2f5a-4733-bea2-d2599a9931f2\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s" Oct 30 19:45:47.702964 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:47.702932 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s" Oct 30 19:45:48.124333 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:48.124302 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s"] Oct 30 19:45:48.125863 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:45:48.125837 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd0da58c_2f5a_4733_bea2_d2599a9931f2.slice/crio-5ca7d8d7a4215876e88336224028efb69936dd960e27dfea2813428be655e368 WatchSource:0}: Error finding container 5ca7d8d7a4215876e88336224028efb69936dd960e27dfea2813428be655e368: Status 404 returned error can't find the container with id 5ca7d8d7a4215876e88336224028efb69936dd960e27dfea2813428be655e368 Oct 30 19:45:48.128126 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:48.128094 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:45:48.128195 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:48.128166 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:45:48.128232 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:48.128194 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:45:48.456141 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:45:48.456060 4292 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-449cc5558c4a: secret "agc-cred-449cc5558c4a" not found Oct 30 19:45:48.456141 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:45:48.456089 4292 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-788b6c6b4d-59ggs: secret "agc-cred-449cc5558c4a" not found Oct 30 19:45:48.456572 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:45:48.456160 4292 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a179efae-1f42-4b73-b2d7-70f1474c3c9f-provider-credentials podName:a179efae-1f42-4b73-b2d7-70f1474c3c9f nodeName:}" failed. No retries permitted until 2026-10-30 19:46:20.456144836 +0000 UTC m=+1283.974460284 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a179efae-1f42-4b73-b2d7-70f1474c3c9f-provider-credentials") pod "payload-processing-external-model-788b6c6b4d-59ggs" (UID: "a179efae-1f42-4b73-b2d7-70f1474c3c9f") : secret "agc-cred-449cc5558c4a" not found Oct 30 19:45:49.002227 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:49.002187 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s" event={"ID":"dd0da58c-2f5a-4733-bea2-d2599a9931f2","Type":"ContainerStarted","Data":"444859bc547d4149a131b2775881f880c3e494ec57fefa5ff319364ff223eb8a"} Oct 30 19:45:49.002445 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:49.002244 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s" event={"ID":"dd0da58c-2f5a-4733-bea2-d2599a9931f2","Type":"ContainerStarted","Data":"5ca7d8d7a4215876e88336224028efb69936dd960e27dfea2813428be655e368"} Oct 30 19:45:49.128100 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:49.128042 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s" podStartSLOduration=2.128026949 podStartE2EDuration="2.128026949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:49.126523951 +0000 UTC m=+1252.644839450" watchObservedRunningTime="2026-10-30 19:45:49.128026949 +0000 UTC m=+1252.646342406" Oct 30 19:45:49.703985 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:49.703952 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s" Oct 30 19:45:49.705417 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:49.705392 4292 patch_prober.go:29] interesting pod/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 19:45:49.705560 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:49.705452 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s" podUID="dd0da58c-2f5a-4733-bea2-d2599a9931f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 19:45:50.703598 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:50.703563 4292 patch_prober.go:29] interesting pod/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 19:45:50.703792 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:50.703614 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s" podUID="dd0da58c-2f5a-4733-bea2-d2599a9931f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 19:45:51.703614 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:51.703581 4292 patch_prober.go:29] interesting pod/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 19:45:51.704021 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:51.703640 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s" podUID="dd0da58c-2f5a-4733-bea2-d2599a9931f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 19:45:52.704162 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:52.704121 4292 patch_prober.go:29] interesting pod/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 19:45:52.704591 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:52.704181 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s" podUID="dd0da58c-2f5a-4733-bea2-d2599a9931f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 19:45:53.703249 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:53.703217 4292 patch_prober.go:29] interesting pod/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 19:45:53.703498 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:53.703271 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s" podUID="dd0da58c-2f5a-4733-bea2-d2599a9931f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 19:45:54.704091 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:54.704052 4292 patch_prober.go:29] interesting pod/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 19:45:54.704538 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:54.704106 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s" podUID="dd0da58c-2f5a-4733-bea2-d2599a9931f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 19:45:55.708437 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:55.708403 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s" Oct 30 19:45:55.708942 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:55.708683 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s" Oct 30 19:45:55.709525 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:55.709506 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s" Oct 30 19:45:56.717563 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:45:56.717532 4292 scope.go:122] "RemoveContainer" containerID="ecb1b79e969368190adebc673e9e94667e6413b22fda42183bb3ea5fe00b05d2" Oct 30 19:46:11.827311 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:11.827277 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-bbfa07/distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2"] Oct 30 19:46:11.827801 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:11.827788 4292 cpu_manager.go:414] "containerMap: removing container" podUID="756dede2-453c-44a3-afa7-b57ccaa0f044" containerName="cleanup" Oct 30 19:46:11.827853 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:11.827802 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="756dede2-453c-44a3-afa7-b57ccaa0f044" containerName="cleanup" Oct 30 19:46:11.827890 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:11.827879 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="756dede2-453c-44a3-afa7-b57ccaa0f044" containerName="cleanup" Oct 30 19:46:11.832734 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:11.832717 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bbfa07/distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2" Oct 30 19:46:11.931597 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:11.931564 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-bbfa07\"/\"distinct-2-w3-bbfa07-kserve-self-signed-certs\"" Oct 30 19:46:11.952775 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:11.952735 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f562f81a-ebd3-4efa-889d-bf4a757527fe-tmp-dir\") pod \"distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2\" (UID: \"f562f81a-ebd3-4efa-889d-bf4a757527fe\") " pod="e2e-models-e2e-worker-w3-bbfa07/distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2" Oct 30 19:46:11.952960 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:11.952802 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f562f81a-ebd3-4efa-889d-bf4a757527fe-dshm\") pod \"distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2\" (UID: \"f562f81a-ebd3-4efa-889d-bf4a757527fe\") " pod="e2e-models-e2e-worker-w3-bbfa07/distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2" Oct 30 19:46:11.952960 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:11.952957 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f562f81a-ebd3-4efa-889d-bf4a757527fe-tls-certs\") pod \"distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2\" (UID: \"f562f81a-ebd3-4efa-889d-bf4a757527fe\") " pod="e2e-models-e2e-worker-w3-bbfa07/distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2" Oct 30 19:46:11.953068 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:11.952983 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpbmq\" (UniqueName: \"kubernetes.io/projected/f562f81a-ebd3-4efa-889d-bf4a757527fe-kube-api-access-bpbmq\") pod \"distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2\" (UID: \"f562f81a-ebd3-4efa-889d-bf4a757527fe\") " pod="e2e-models-e2e-worker-w3-bbfa07/distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2" Oct 30 19:46:11.953068 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:11.953008 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f562f81a-ebd3-4efa-889d-bf4a757527fe-home\") pod \"distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2\" (UID: \"f562f81a-ebd3-4efa-889d-bf4a757527fe\") " pod="e2e-models-e2e-worker-w3-bbfa07/distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2" Oct 30 19:46:11.953144 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:11.953098 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f562f81a-ebd3-4efa-889d-bf4a757527fe-model-cache\") pod \"distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2\" (UID: \"f562f81a-ebd3-4efa-889d-bf4a757527fe\") " pod="e2e-models-e2e-worker-w3-bbfa07/distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2" Oct 30 19:46:12.010769 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:12.010725 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-bbfa07/distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2"] Oct 30 19:46:12.054431 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:12.054395 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f562f81a-ebd3-4efa-889d-bf4a757527fe-model-cache\") pod \"distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2\" (UID: \"f562f81a-ebd3-4efa-889d-bf4a757527fe\") " pod="e2e-models-e2e-worker-w3-bbfa07/distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2" Oct 30 19:46:12.054643 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:12.054475 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f562f81a-ebd3-4efa-889d-bf4a757527fe-tmp-dir\") pod \"distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2\" (UID: \"f562f81a-ebd3-4efa-889d-bf4a757527fe\") " pod="e2e-models-e2e-worker-w3-bbfa07/distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2" Oct 30 19:46:12.054643 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:12.054531 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f562f81a-ebd3-4efa-889d-bf4a757527fe-dshm\") pod \"distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2\" (UID: \"f562f81a-ebd3-4efa-889d-bf4a757527fe\") " pod="e2e-models-e2e-worker-w3-bbfa07/distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2" Oct 30 19:46:12.054761 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:12.054647 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f562f81a-ebd3-4efa-889d-bf4a757527fe-tls-certs\") pod \"distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2\" (UID: \"f562f81a-ebd3-4efa-889d-bf4a757527fe\") " pod="e2e-models-e2e-worker-w3-bbfa07/distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2" Oct 30 19:46:12.054761 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:12.054684 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpbmq\" (UniqueName: \"kubernetes.io/projected/f562f81a-ebd3-4efa-889d-bf4a757527fe-kube-api-access-bpbmq\") pod \"distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2\" (UID: \"f562f81a-ebd3-4efa-889d-bf4a757527fe\") " pod="e2e-models-e2e-worker-w3-bbfa07/distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2" Oct 30 19:46:12.054761 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:12.054722 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f562f81a-ebd3-4efa-889d-bf4a757527fe-home\") pod \"distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2\" (UID: \"f562f81a-ebd3-4efa-889d-bf4a757527fe\") " pod="e2e-models-e2e-worker-w3-bbfa07/distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2" Oct 30 19:46:12.054921 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:12.054817 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f562f81a-ebd3-4efa-889d-bf4a757527fe-model-cache\") pod \"distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2\" (UID: \"f562f81a-ebd3-4efa-889d-bf4a757527fe\") " pod="e2e-models-e2e-worker-w3-bbfa07/distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2" Oct 30 19:46:12.055035 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:12.055010 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f562f81a-ebd3-4efa-889d-bf4a757527fe-home\") pod \"distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2\" (UID: \"f562f81a-ebd3-4efa-889d-bf4a757527fe\") " pod="e2e-models-e2e-worker-w3-bbfa07/distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2" Oct 30 19:46:12.055503 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:12.055452 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f562f81a-ebd3-4efa-889d-bf4a757527fe-tmp-dir\") pod \"distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2\" (UID: \"f562f81a-ebd3-4efa-889d-bf4a757527fe\") " pod="e2e-models-e2e-worker-w3-bbfa07/distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2" Oct 30 19:46:12.057866 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:12.057840 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f562f81a-ebd3-4efa-889d-bf4a757527fe-dshm\") pod \"distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2\" (UID: \"f562f81a-ebd3-4efa-889d-bf4a757527fe\") " pod="e2e-models-e2e-worker-w3-bbfa07/distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2" Oct 30 19:46:12.058185 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:12.058166 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f562f81a-ebd3-4efa-889d-bf4a757527fe-tls-certs\") pod \"distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2\" (UID: \"f562f81a-ebd3-4efa-889d-bf4a757527fe\") " pod="e2e-models-e2e-worker-w3-bbfa07/distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2" Oct 30 19:46:12.116044 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:12.115958 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpbmq\" (UniqueName: \"kubernetes.io/projected/f562f81a-ebd3-4efa-889d-bf4a757527fe-kube-api-access-bpbmq\") pod \"distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2\" (UID: \"f562f81a-ebd3-4efa-889d-bf4a757527fe\") " pod="e2e-models-e2e-worker-w3-bbfa07/distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2" Oct 30 19:46:12.144569 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:12.144536 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bbfa07/distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2" Oct 30 19:46:12.639154 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:46:12.639114 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf562f81a_ebd3_4efa_889d_bf4a757527fe.slice/crio-112003ebbabc11c24b96769f7af9ea3d5a84f94d674fbe644ba38b75ab9bc68f WatchSource:0}: Error finding container 112003ebbabc11c24b96769f7af9ea3d5a84f94d674fbe644ba38b75ab9bc68f: Status 404 returned error can't find the container with id 112003ebbabc11c24b96769f7af9ea3d5a84f94d674fbe644ba38b75ab9bc68f Oct 30 19:46:12.639283 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:12.639198 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-bbfa07/distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2"] Oct 30 19:46:13.133577 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:13.133537 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bbfa07/distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2" event={"ID":"f562f81a-ebd3-4efa-889d-bf4a757527fe","Type":"ContainerStarted","Data":"e3959c82e66eb88d22e421cd2eb52089348bee2b84576f26f7e5b87c20862b59"} Oct 30 19:46:13.133577 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:13.133580 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bbfa07/distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2" event={"ID":"f562f81a-ebd3-4efa-889d-bf4a757527fe","Type":"ContainerStarted","Data":"112003ebbabc11c24b96769f7af9ea3d5a84f94d674fbe644ba38b75ab9bc68f"} Oct 30 19:46:13.144851 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:13.144819 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bbfa07/distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2" Oct 30 19:46:13.221816 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:46:13.221782 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:46:13.247720 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:13.247668 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-bbfa07/distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2" podStartSLOduration=2.24765387 podStartE2EDuration="2.24765387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:13.245902329 +0000 UTC m=+1276.764217801" watchObservedRunningTime="2026-10-30 19:46:13.24765387 +0000 UTC m=+1276.765969327" Oct 30 19:46:20.484075 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:46:20.484040 4292 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-449cc5558c4a: secret "agc-cred-449cc5558c4a" not found Oct 30 19:46:20.484075 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:46:20.484068 4292 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-788b6c6b4d-59ggs: secret "agc-cred-449cc5558c4a" not found Oct 30 19:46:20.484568 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:46:20.484122 4292 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a179efae-1f42-4b73-b2d7-70f1474c3c9f-provider-credentials podName:a179efae-1f42-4b73-b2d7-70f1474c3c9f nodeName:}" failed. No retries permitted until 2026-10-30 19:47:24.484106175 +0000 UTC m=+1348.002421624 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a179efae-1f42-4b73-b2d7-70f1474c3c9f-provider-credentials") pod "payload-processing-external-model-788b6c6b4d-59ggs" (UID: "a179efae-1f42-4b73-b2d7-70f1474c3c9f") : secret "agc-cred-449cc5558c4a" not found Oct 30 19:46:22.145467 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:22.145426 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-bbfa07/distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2" Oct 30 19:46:22.158153 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:22.158118 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-bbfa07/distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2" Oct 30 19:46:22.192508 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:22.192476 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bbfa07/distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2" Oct 30 19:46:35.185044 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:35.184999 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f"] Oct 30 19:46:35.190242 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:35.190217 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f" Oct 30 19:46:35.243553 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:35.243518 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f"] Oct 30 19:46:35.252404 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:35.252369 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3845b969-7d5b-48d4-a2d8-dfbe93f91c05-workload-certs\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f\" (UID: \"3845b969-7d5b-48d4-a2d8-dfbe93f91c05\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f" Oct 30 19:46:35.252585 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:35.252445 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3845b969-7d5b-48d4-a2d8-dfbe93f91c05-istio-token\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f\" (UID: \"3845b969-7d5b-48d4-a2d8-dfbe93f91c05\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f" Oct 30 19:46:35.252585 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:35.252477 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3845b969-7d5b-48d4-a2d8-dfbe93f91c05-istio-envoy\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f\" (UID: \"3845b969-7d5b-48d4-a2d8-dfbe93f91c05\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f" Oct 30 19:46:35.252585 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:35.252509 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3845b969-7d5b-48d4-a2d8-dfbe93f91c05-istio-data\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f\" (UID: \"3845b969-7d5b-48d4-a2d8-dfbe93f91c05\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f" Oct 30 19:46:35.252759 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:35.252601 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52q4m\" (UniqueName: \"kubernetes.io/projected/3845b969-7d5b-48d4-a2d8-dfbe93f91c05-kube-api-access-52q4m\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f\" (UID: \"3845b969-7d5b-48d4-a2d8-dfbe93f91c05\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f" Oct 30 19:46:35.252759 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:35.252703 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3845b969-7d5b-48d4-a2d8-dfbe93f91c05-workload-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f\" (UID: \"3845b969-7d5b-48d4-a2d8-dfbe93f91c05\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f" Oct 30 19:46:35.252759 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:35.252732 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3845b969-7d5b-48d4-a2d8-dfbe93f91c05-credential-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f\" (UID: \"3845b969-7d5b-48d4-a2d8-dfbe93f91c05\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f" Oct 30 19:46:35.252914 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:35.252756 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3845b969-7d5b-48d4-a2d8-dfbe93f91c05-istiod-ca-cert\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f\" (UID: \"3845b969-7d5b-48d4-a2d8-dfbe93f91c05\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f" Oct 30 19:46:35.252914 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:35.252814 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3845b969-7d5b-48d4-a2d8-dfbe93f91c05-istio-podinfo\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f\" (UID: \"3845b969-7d5b-48d4-a2d8-dfbe93f91c05\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f" Oct 30 19:46:35.290832 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:35.290798 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-98a19b-openshift-default-dockercfg-rhphv\"" Oct 30 19:46:35.353694 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:35.353651 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52q4m\" (UniqueName: \"kubernetes.io/projected/3845b969-7d5b-48d4-a2d8-dfbe93f91c05-kube-api-access-52q4m\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f\" (UID: \"3845b969-7d5b-48d4-a2d8-dfbe93f91c05\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f" Oct 30 19:46:35.353897 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:35.353754 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3845b969-7d5b-48d4-a2d8-dfbe93f91c05-workload-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f\" (UID: \"3845b969-7d5b-48d4-a2d8-dfbe93f91c05\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f" Oct 30 19:46:35.353897 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:35.353782 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3845b969-7d5b-48d4-a2d8-dfbe93f91c05-credential-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f\" (UID: \"3845b969-7d5b-48d4-a2d8-dfbe93f91c05\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f" Oct 30 19:46:35.353897 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:35.353803 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3845b969-7d5b-48d4-a2d8-dfbe93f91c05-istiod-ca-cert\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f\" (UID: \"3845b969-7d5b-48d4-a2d8-dfbe93f91c05\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f" Oct 30 19:46:35.354073 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:35.354011 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3845b969-7d5b-48d4-a2d8-dfbe93f91c05-istio-podinfo\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f\" (UID: \"3845b969-7d5b-48d4-a2d8-dfbe93f91c05\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f" Oct 30 19:46:35.354126 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:35.354077 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3845b969-7d5b-48d4-a2d8-dfbe93f91c05-workload-certs\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f\" (UID: \"3845b969-7d5b-48d4-a2d8-dfbe93f91c05\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f" Oct 30 19:46:35.354177 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:35.354144 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3845b969-7d5b-48d4-a2d8-dfbe93f91c05-istio-token\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f\" (UID: \"3845b969-7d5b-48d4-a2d8-dfbe93f91c05\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f" Oct 30 19:46:35.354235 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:35.354182 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3845b969-7d5b-48d4-a2d8-dfbe93f91c05-istio-envoy\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f\" (UID: \"3845b969-7d5b-48d4-a2d8-dfbe93f91c05\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f" Oct 30 19:46:35.354235 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:35.354218 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3845b969-7d5b-48d4-a2d8-dfbe93f91c05-istio-data\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f\" (UID: \"3845b969-7d5b-48d4-a2d8-dfbe93f91c05\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f" Oct 30 19:46:35.354332 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:35.354271 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3845b969-7d5b-48d4-a2d8-dfbe93f91c05-workload-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f\" (UID: \"3845b969-7d5b-48d4-a2d8-dfbe93f91c05\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f" Oct 30 19:46:35.354332 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:35.354298 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3845b969-7d5b-48d4-a2d8-dfbe93f91c05-credential-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f\" (UID: \"3845b969-7d5b-48d4-a2d8-dfbe93f91c05\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f" Oct 30 19:46:35.355000 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:35.354970 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3845b969-7d5b-48d4-a2d8-dfbe93f91c05-istiod-ca-cert\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f\" (UID: \"3845b969-7d5b-48d4-a2d8-dfbe93f91c05\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f" Oct 30 19:46:35.355508 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:35.355162 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3845b969-7d5b-48d4-a2d8-dfbe93f91c05-workload-certs\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f\" (UID: \"3845b969-7d5b-48d4-a2d8-dfbe93f91c05\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f" Oct 30 19:46:35.355508 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:35.355206 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3845b969-7d5b-48d4-a2d8-dfbe93f91c05-istio-data\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f\" (UID: \"3845b969-7d5b-48d4-a2d8-dfbe93f91c05\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f" Oct 30 19:46:35.356899 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:35.356873 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3845b969-7d5b-48d4-a2d8-dfbe93f91c05-istio-podinfo\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f\" (UID: \"3845b969-7d5b-48d4-a2d8-dfbe93f91c05\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f" Oct 30 19:46:35.357270 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:35.357245 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3845b969-7d5b-48d4-a2d8-dfbe93f91c05-istio-envoy\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f\" (UID: \"3845b969-7d5b-48d4-a2d8-dfbe93f91c05\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f" Oct 30 19:46:35.416974 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:35.416934 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3845b969-7d5b-48d4-a2d8-dfbe93f91c05-istio-token\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f\" (UID: \"3845b969-7d5b-48d4-a2d8-dfbe93f91c05\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f" Oct 30 19:46:35.418118 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:35.418093 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52q4m\" (UniqueName: \"kubernetes.io/projected/3845b969-7d5b-48d4-a2d8-dfbe93f91c05-kube-api-access-52q4m\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f\" (UID: \"3845b969-7d5b-48d4-a2d8-dfbe93f91c05\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f" Oct 30 19:46:35.505512 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:35.505423 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f" Oct 30 19:46:36.001863 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:36.001830 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f"] Oct 30 19:46:36.002803 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:46:36.002776 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3845b969_7d5b_48d4_a2d8_dfbe93f91c05.slice/crio-ade2d2b271557a9e48b069568b2b81beb502947e4e950c1afbceace2768a045d WatchSource:0}: Error finding container ade2d2b271557a9e48b069568b2b81beb502947e4e950c1afbceace2768a045d: Status 404 returned error can't find the container with id ade2d2b271557a9e48b069568b2b81beb502947e4e950c1afbceace2768a045d Oct 30 19:46:36.005494 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:36.005460 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:46:36.005566 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:36.005532 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:46:36.005566 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:36.005561 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:46:36.256952 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:36.256863 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f" event={"ID":"3845b969-7d5b-48d4-a2d8-dfbe93f91c05","Type":"ContainerStarted","Data":"0470224b914e5efac94b13fd57065c9111b37a70a940942abe7b68b24b4dc94a"} Oct 30 19:46:36.256952 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:36.256924 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f" event={"ID":"3845b969-7d5b-48d4-a2d8-dfbe93f91c05","Type":"ContainerStarted","Data":"ade2d2b271557a9e48b069568b2b81beb502947e4e950c1afbceace2768a045d"} Oct 30 19:46:36.431644 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:36.431579 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f" podStartSLOduration=1.431557969 podStartE2EDuration="1.431557969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:36.380257785 +0000 UTC m=+1299.898573260" watchObservedRunningTime="2026-10-30 19:46:36.431557969 +0000 UTC m=+1299.949873428" Oct 30 19:46:36.505744 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:36.505703 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f" Oct 30 19:46:37.507740 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:37.507697 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 30 19:46:37.508137 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:37.507764 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f" podUID="3845b969-7d5b-48d4-a2d8-dfbe93f91c05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 30 19:46:38.506146 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:38.506113 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 30 19:46:38.506330 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:38.506165 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f" podUID="3845b969-7d5b-48d4-a2d8-dfbe93f91c05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 30 19:46:39.506734 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:39.506699 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 30 19:46:39.507160 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:39.506754 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f" podUID="3845b969-7d5b-48d4-a2d8-dfbe93f91c05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 30 19:46:40.506087 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:40.506003 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 30 19:46:40.506319 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:40.506298 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f" podUID="3845b969-7d5b-48d4-a2d8-dfbe93f91c05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 30 19:46:41.506209 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:41.506175 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 30 19:46:41.506718 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:41.506232 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f" podUID="3845b969-7d5b-48d4-a2d8-dfbe93f91c05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 30 19:46:42.506223 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:42.506183 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 30 19:46:42.506645 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:42.506238 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f" podUID="3845b969-7d5b-48d4-a2d8-dfbe93f91c05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 30 19:46:43.220724 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:46:43.220680 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:46:43.511141 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:43.511059 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f" Oct 30 19:46:43.511633 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:43.511449 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f" Oct 30 19:46:43.512241 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:43.512218 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f" Oct 30 19:46:45.408592 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:45.408552 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-b0a183/e2e-conflict-a-c338110a-kserve-c7dc6d695-7v6hl"] Oct 30 19:46:57.529068 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:46:57.529025 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-788b6c6b4d-59ggs"] Oct 30 19:47:00.431269 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:00.431228 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-b0a183/e2e-conflict-a-c338110a-kserve-c7dc6d695-7v6hl" podUID="cc2aacd3-0456-4c2b-8d5e-3743ec9d872e" containerName="main" containerID="cri-o://0cb68409f2e2a127f7fbb1542bd6e7fd0a8da215235f9425b84c72579f1f63d3" gracePeriod=45 Oct 30 19:47:00.686228 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:00.686156 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b0a183/e2e-conflict-a-c338110a-kserve-c7dc6d695-7v6hl" Oct 30 19:47:00.776380 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:00.776309 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc2aacd3-0456-4c2b-8d5e-3743ec9d872e-kube-api-access-ctpqp\" (UniqueName: \"kubernetes.io/projected/cc2aacd3-0456-4c2b-8d5e-3743ec9d872e-kube-api-access-ctpqp\") pod \"cc2aacd3-0456-4c2b-8d5e-3743ec9d872e\" (UID: \"cc2aacd3-0456-4c2b-8d5e-3743ec9d872e\") " Oct 30 19:47:00.776380 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:00.776385 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc2aacd3-0456-4c2b-8d5e-3743ec9d872e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc2aacd3-0456-4c2b-8d5e-3743ec9d872e-model-cache\") pod \"cc2aacd3-0456-4c2b-8d5e-3743ec9d872e\" (UID: \"cc2aacd3-0456-4c2b-8d5e-3743ec9d872e\") " Oct 30 19:47:00.776667 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:00.776458 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc2aacd3-0456-4c2b-8d5e-3743ec9d872e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc2aacd3-0456-4c2b-8d5e-3743ec9d872e-dshm\") pod \"cc2aacd3-0456-4c2b-8d5e-3743ec9d872e\" (UID: \"cc2aacd3-0456-4c2b-8d5e-3743ec9d872e\") " Oct 30 19:47:00.776667 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:00.776523 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc2aacd3-0456-4c2b-8d5e-3743ec9d872e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc2aacd3-0456-4c2b-8d5e-3743ec9d872e-tmp-dir\") pod \"cc2aacd3-0456-4c2b-8d5e-3743ec9d872e\" (UID: \"cc2aacd3-0456-4c2b-8d5e-3743ec9d872e\") " Oct 30 19:47:00.776667 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:00.776553 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc2aacd3-0456-4c2b-8d5e-3743ec9d872e-tls-certs\" (UniqueName: \"kubernetes.io/secret/cc2aacd3-0456-4c2b-8d5e-3743ec9d872e-tls-certs\") pod \"cc2aacd3-0456-4c2b-8d5e-3743ec9d872e\" (UID: \"cc2aacd3-0456-4c2b-8d5e-3743ec9d872e\") " Oct 30 19:47:00.776667 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:00.776588 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc2aacd3-0456-4c2b-8d5e-3743ec9d872e-home\" (UniqueName: \"kubernetes.io/empty-dir/cc2aacd3-0456-4c2b-8d5e-3743ec9d872e-home\") pod \"cc2aacd3-0456-4c2b-8d5e-3743ec9d872e\" (UID: \"cc2aacd3-0456-4c2b-8d5e-3743ec9d872e\") " Oct 30 19:47:00.776872 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:00.776721 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc2aacd3-0456-4c2b-8d5e-3743ec9d872e-model-cache" pod "cc2aacd3-0456-4c2b-8d5e-3743ec9d872e" (UID: "cc2aacd3-0456-4c2b-8d5e-3743ec9d872e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:00.776872 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:00.776810 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc2aacd3-0456-4c2b-8d5e-3743ec9d872e-tmp-dir" pod "cc2aacd3-0456-4c2b-8d5e-3743ec9d872e" (UID: "cc2aacd3-0456-4c2b-8d5e-3743ec9d872e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:00.776985 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:00.776878 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc2aacd3-0456-4c2b-8d5e-3743ec9d872e-home" pod "cc2aacd3-0456-4c2b-8d5e-3743ec9d872e" (UID: "cc2aacd3-0456-4c2b-8d5e-3743ec9d872e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:00.777116 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:00.777095 4292 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc2aacd3-0456-4c2b-8d5e-3743ec9d872e-model-cache\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:00.777180 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:00.777121 4292 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc2aacd3-0456-4c2b-8d5e-3743ec9d872e-tmp-dir\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:00.777180 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:00.777134 4292 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc2aacd3-0456-4c2b-8d5e-3743ec9d872e-home\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:00.778731 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:00.778702 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc2aacd3-0456-4c2b-8d5e-3743ec9d872e-dshm" pod "cc2aacd3-0456-4c2b-8d5e-3743ec9d872e" (UID: "cc2aacd3-0456-4c2b-8d5e-3743ec9d872e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:00.778842 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:00.778761 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc2aacd3-0456-4c2b-8d5e-3743ec9d872e-kube-api-access-ctpqp" pod "cc2aacd3-0456-4c2b-8d5e-3743ec9d872e" (UID: "cc2aacd3-0456-4c2b-8d5e-3743ec9d872e"). InnerVolumeSpecName "kube-api-access-ctpqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:47:00.778842 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:00.778778 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc2aacd3-0456-4c2b-8d5e-3743ec9d872e-tls-certs" pod "cc2aacd3-0456-4c2b-8d5e-3743ec9d872e" (UID: "cc2aacd3-0456-4c2b-8d5e-3743ec9d872e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:47:00.878580 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:00.878545 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctpqp\" (UniqueName: \"kubernetes.io/projected/cc2aacd3-0456-4c2b-8d5e-3743ec9d872e-kube-api-access-ctpqp\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:00.878580 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:00.878574 4292 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc2aacd3-0456-4c2b-8d5e-3743ec9d872e-dshm\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:00.878580 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:00.878583 4292 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc2aacd3-0456-4c2b-8d5e-3743ec9d872e-tls-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:01.429601 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:01.429562 4292 generic.go:356] "Generic (PLEG): container finished" podID="cc2aacd3-0456-4c2b-8d5e-3743ec9d872e" containerID="0cb68409f2e2a127f7fbb1542bd6e7fd0a8da215235f9425b84c72579f1f63d3" exitCode=0 Oct 30 19:47:01.429601 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:01.429605 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b0a183/e2e-conflict-a-c338110a-kserve-c7dc6d695-7v6hl" Oct 30 19:47:01.429875 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:01.429620 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b0a183/e2e-conflict-a-c338110a-kserve-c7dc6d695-7v6hl" event={"ID":"cc2aacd3-0456-4c2b-8d5e-3743ec9d872e","Type":"ContainerDied","Data":"0cb68409f2e2a127f7fbb1542bd6e7fd0a8da215235f9425b84c72579f1f63d3"} Oct 30 19:47:01.429875 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:01.429660 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b0a183/e2e-conflict-a-c338110a-kserve-c7dc6d695-7v6hl" event={"ID":"cc2aacd3-0456-4c2b-8d5e-3743ec9d872e","Type":"ContainerDied","Data":"ca67dc9952c86a8b2c1d10f275a635d170c25882d877ec5f5eaceb9aa7841376"} Oct 30 19:47:01.429875 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:01.429684 4292 scope.go:122] "RemoveContainer" containerID="0cb68409f2e2a127f7fbb1542bd6e7fd0a8da215235f9425b84c72579f1f63d3" Oct 30 19:47:01.439535 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:01.439302 4292 scope.go:122] "RemoveContainer" containerID="0cb68409f2e2a127f7fbb1542bd6e7fd0a8da215235f9425b84c72579f1f63d3" Oct 30 19:47:01.439768 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:47:01.439578 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cb68409f2e2a127f7fbb1542bd6e7fd0a8da215235f9425b84c72579f1f63d3\": container with ID starting with 0cb68409f2e2a127f7fbb1542bd6e7fd0a8da215235f9425b84c72579f1f63d3 not found: ID does not exist" containerID="0cb68409f2e2a127f7fbb1542bd6e7fd0a8da215235f9425b84c72579f1f63d3" Oct 30 19:47:01.439768 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:01.439604 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cb68409f2e2a127f7fbb1542bd6e7fd0a8da215235f9425b84c72579f1f63d3"} err="failed to get container status \"0cb68409f2e2a127f7fbb1542bd6e7fd0a8da215235f9425b84c72579f1f63d3\": rpc error: code = NotFound desc = could not find container \"0cb68409f2e2a127f7fbb1542bd6e7fd0a8da215235f9425b84c72579f1f63d3\": container with ID starting with 0cb68409f2e2a127f7fbb1542bd6e7fd0a8da215235f9425b84c72579f1f63d3 not found: ID does not exist" Oct 30 19:47:01.598606 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:01.598566 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-b0a183/e2e-conflict-a-c338110a-kserve-c7dc6d695-7v6hl"] Oct 30 19:47:01.648788 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:01.648750 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-b0a183/e2e-conflict-a-c338110a-kserve-c7dc6d695-7v6hl"] Oct 30 19:47:02.530198 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:02.530136 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-788b6c6b4d-59ggs" podUID="a179efae-1f42-4b73-b2d7-70f1474c3c9f" containerName="payload-processing" containerID="cri-o://9472ee5955ca5a2b3ac7bfcd49677bc482125e3434d2cb3c61dee8b50e4969cf" gracePeriod=25 Oct 30 19:47:02.582749 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:02.582718 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc2aacd3-0456-4c2b-8d5e-3743ec9d872e" path="/var/lib/kubelet/pods/cc2aacd3-0456-4c2b-8d5e-3743ec9d872e/volumes" Oct 30 19:47:02.765048 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:02.765014 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-a74d15/distinct-2-w0-a74d15-kserve-846bdb9948-ht54r"] Oct 30 19:47:02.765528 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:02.765511 4292 cpu_manager.go:414] "containerMap: removing container" podUID="cc2aacd3-0456-4c2b-8d5e-3743ec9d872e" containerName="main" Oct 30 19:47:02.765588 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:02.765529 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc2aacd3-0456-4c2b-8d5e-3743ec9d872e" containerName="main" Oct 30 19:47:02.765663 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:02.765650 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc2aacd3-0456-4c2b-8d5e-3743ec9d872e" containerName="main" Oct 30 19:47:02.769072 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:02.769054 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a74d15/distinct-2-w0-a74d15-kserve-846bdb9948-ht54r" Oct 30 19:47:02.797016 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:02.796526 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-788b6c6b4d-59ggs" Oct 30 19:47:02.824665 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:02.824633 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a74d15/distinct-2-w0-a74d15-kserve-846bdb9948-ht54r"] Oct 30 19:47:02.872971 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:02.872933 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a74d15\"/\"distinct-2-w0-a74d15-kserve-self-signed-certs\"" Oct 30 19:47:02.902305 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:02.902272 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a179efae-1f42-4b73-b2d7-70f1474c3c9f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a179efae-1f42-4b73-b2d7-70f1474c3c9f-provider-credentials\") pod \"a179efae-1f42-4b73-b2d7-70f1474c3c9f\" (UID: \"a179efae-1f42-4b73-b2d7-70f1474c3c9f\") " Oct 30 19:47:02.902533 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:02.902326 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a179efae-1f42-4b73-b2d7-70f1474c3c9f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a179efae-1f42-4b73-b2d7-70f1474c3c9f-plugins-config-volume\") pod \"a179efae-1f42-4b73-b2d7-70f1474c3c9f\" (UID: \"a179efae-1f42-4b73-b2d7-70f1474c3c9f\") " Oct 30 19:47:02.902533 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:02.902433 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a179efae-1f42-4b73-b2d7-70f1474c3c9f-kube-api-access-t2dqr\" (UniqueName: \"kubernetes.io/projected/a179efae-1f42-4b73-b2d7-70f1474c3c9f-kube-api-access-t2dqr\") pod \"a179efae-1f42-4b73-b2d7-70f1474c3c9f\" (UID: \"a179efae-1f42-4b73-b2d7-70f1474c3c9f\") " Oct 30 19:47:02.902533 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:02.902514 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a179efae-1f42-4b73-b2d7-70f1474c3c9f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a179efae-1f42-4b73-b2d7-70f1474c3c9f-routing-overlay-volume\") pod \"a179efae-1f42-4b73-b2d7-70f1474c3c9f\" (UID: \"a179efae-1f42-4b73-b2d7-70f1474c3c9f\") " Oct 30 19:47:02.902731 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:02.902712 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f70232fa-7106-4ec6-8c8e-277c4ce4c719-dshm\") pod \"distinct-2-w0-a74d15-kserve-846bdb9948-ht54r\" (UID: \"f70232fa-7106-4ec6-8c8e-277c4ce4c719\") " pod="e2e-models-e2e-worker-w0-a74d15/distinct-2-w0-a74d15-kserve-846bdb9948-ht54r" Oct 30 19:47:02.902797 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:02.902768 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f70232fa-7106-4ec6-8c8e-277c4ce4c719-tls-certs\") pod \"distinct-2-w0-a74d15-kserve-846bdb9948-ht54r\" (UID: \"f70232fa-7106-4ec6-8c8e-277c4ce4c719\") " pod="e2e-models-e2e-worker-w0-a74d15/distinct-2-w0-a74d15-kserve-846bdb9948-ht54r" Oct 30 19:47:02.902895 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:02.902869 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a179efae-1f42-4b73-b2d7-70f1474c3c9f-plugins-config-volume" pod "a179efae-1f42-4b73-b2d7-70f1474c3c9f" (UID: "a179efae-1f42-4b73-b2d7-70f1474c3c9f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:47:02.902953 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:02.902888 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a179efae-1f42-4b73-b2d7-70f1474c3c9f-routing-overlay-volume" pod "a179efae-1f42-4b73-b2d7-70f1474c3c9f" (UID: "a179efae-1f42-4b73-b2d7-70f1474c3c9f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:47:02.902953 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:02.902906 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f70232fa-7106-4ec6-8c8e-277c4ce4c719-home\") pod \"distinct-2-w0-a74d15-kserve-846bdb9948-ht54r\" (UID: \"f70232fa-7106-4ec6-8c8e-277c4ce4c719\") " pod="e2e-models-e2e-worker-w0-a74d15/distinct-2-w0-a74d15-kserve-846bdb9948-ht54r" Oct 30 19:47:02.903062 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:02.903043 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f70232fa-7106-4ec6-8c8e-277c4ce4c719-model-cache\") pod \"distinct-2-w0-a74d15-kserve-846bdb9948-ht54r\" (UID: \"f70232fa-7106-4ec6-8c8e-277c4ce4c719\") " pod="e2e-models-e2e-worker-w0-a74d15/distinct-2-w0-a74d15-kserve-846bdb9948-ht54r" Oct 30 19:47:02.903110 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:02.903096 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f70232fa-7106-4ec6-8c8e-277c4ce4c719-tmp-dir\") pod \"distinct-2-w0-a74d15-kserve-846bdb9948-ht54r\" (UID: \"f70232fa-7106-4ec6-8c8e-277c4ce4c719\") " pod="e2e-models-e2e-worker-w0-a74d15/distinct-2-w0-a74d15-kserve-846bdb9948-ht54r" Oct 30 19:47:02.903162 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:02.903146 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r248s\" (UniqueName: \"kubernetes.io/projected/f70232fa-7106-4ec6-8c8e-277c4ce4c719-kube-api-access-r248s\") pod \"distinct-2-w0-a74d15-kserve-846bdb9948-ht54r\" (UID: \"f70232fa-7106-4ec6-8c8e-277c4ce4c719\") " pod="e2e-models-e2e-worker-w0-a74d15/distinct-2-w0-a74d15-kserve-846bdb9948-ht54r" Oct 30 19:47:02.903294 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:02.903282 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a179efae-1f42-4b73-b2d7-70f1474c3c9f-plugins-config-volume\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:02.903331 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:02.903297 4292 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a179efae-1f42-4b73-b2d7-70f1474c3c9f-routing-overlay-volume\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:02.904777 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:02.904747 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a179efae-1f42-4b73-b2d7-70f1474c3c9f-kube-api-access-t2dqr" pod "a179efae-1f42-4b73-b2d7-70f1474c3c9f" (UID: "a179efae-1f42-4b73-b2d7-70f1474c3c9f"). InnerVolumeSpecName "kube-api-access-t2dqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:47:02.904777 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:02.904753 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a179efae-1f42-4b73-b2d7-70f1474c3c9f-provider-credentials" pod "a179efae-1f42-4b73-b2d7-70f1474c3c9f" (UID: "a179efae-1f42-4b73-b2d7-70f1474c3c9f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:47:03.006904 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:03.004057 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r248s\" (UniqueName: \"kubernetes.io/projected/f70232fa-7106-4ec6-8c8e-277c4ce4c719-kube-api-access-r248s\") pod \"distinct-2-w0-a74d15-kserve-846bdb9948-ht54r\" (UID: \"f70232fa-7106-4ec6-8c8e-277c4ce4c719\") " pod="e2e-models-e2e-worker-w0-a74d15/distinct-2-w0-a74d15-kserve-846bdb9948-ht54r" Oct 30 19:47:03.006904 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:03.004602 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f70232fa-7106-4ec6-8c8e-277c4ce4c719-dshm\") pod \"distinct-2-w0-a74d15-kserve-846bdb9948-ht54r\" (UID: \"f70232fa-7106-4ec6-8c8e-277c4ce4c719\") " pod="e2e-models-e2e-worker-w0-a74d15/distinct-2-w0-a74d15-kserve-846bdb9948-ht54r" Oct 30 19:47:03.006904 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:03.004662 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f70232fa-7106-4ec6-8c8e-277c4ce4c719-tls-certs\") pod \"distinct-2-w0-a74d15-kserve-846bdb9948-ht54r\" (UID: \"f70232fa-7106-4ec6-8c8e-277c4ce4c719\") " pod="e2e-models-e2e-worker-w0-a74d15/distinct-2-w0-a74d15-kserve-846bdb9948-ht54r" Oct 30 19:47:03.006904 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:03.004709 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f70232fa-7106-4ec6-8c8e-277c4ce4c719-home\") pod \"distinct-2-w0-a74d15-kserve-846bdb9948-ht54r\" (UID: \"f70232fa-7106-4ec6-8c8e-277c4ce4c719\") " pod="e2e-models-e2e-worker-w0-a74d15/distinct-2-w0-a74d15-kserve-846bdb9948-ht54r" Oct 30 19:47:03.006904 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:03.004782 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f70232fa-7106-4ec6-8c8e-277c4ce4c719-model-cache\") pod \"distinct-2-w0-a74d15-kserve-846bdb9948-ht54r\" (UID: \"f70232fa-7106-4ec6-8c8e-277c4ce4c719\") " pod="e2e-models-e2e-worker-w0-a74d15/distinct-2-w0-a74d15-kserve-846bdb9948-ht54r" Oct 30 19:47:03.006904 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:03.004840 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f70232fa-7106-4ec6-8c8e-277c4ce4c719-tmp-dir\") pod \"distinct-2-w0-a74d15-kserve-846bdb9948-ht54r\" (UID: \"f70232fa-7106-4ec6-8c8e-277c4ce4c719\") " pod="e2e-models-e2e-worker-w0-a74d15/distinct-2-w0-a74d15-kserve-846bdb9948-ht54r" Oct 30 19:47:03.006904 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:03.004903 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2dqr\" (UniqueName: \"kubernetes.io/projected/a179efae-1f42-4b73-b2d7-70f1474c3c9f-kube-api-access-t2dqr\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:03.006904 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:03.004921 4292 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a179efae-1f42-4b73-b2d7-70f1474c3c9f-provider-credentials\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:03.006904 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:03.005308 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f70232fa-7106-4ec6-8c8e-277c4ce4c719-tmp-dir\") pod \"distinct-2-w0-a74d15-kserve-846bdb9948-ht54r\" (UID: \"f70232fa-7106-4ec6-8c8e-277c4ce4c719\") " pod="e2e-models-e2e-worker-w0-a74d15/distinct-2-w0-a74d15-kserve-846bdb9948-ht54r" Oct 30 19:47:03.006904 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:03.005397 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f70232fa-7106-4ec6-8c8e-277c4ce4c719-home\") pod \"distinct-2-w0-a74d15-kserve-846bdb9948-ht54r\" (UID: \"f70232fa-7106-4ec6-8c8e-277c4ce4c719\") " pod="e2e-models-e2e-worker-w0-a74d15/distinct-2-w0-a74d15-kserve-846bdb9948-ht54r" Oct 30 19:47:03.006904 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:03.005587 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f70232fa-7106-4ec6-8c8e-277c4ce4c719-model-cache\") pod \"distinct-2-w0-a74d15-kserve-846bdb9948-ht54r\" (UID: \"f70232fa-7106-4ec6-8c8e-277c4ce4c719\") " pod="e2e-models-e2e-worker-w0-a74d15/distinct-2-w0-a74d15-kserve-846bdb9948-ht54r" Oct 30 19:47:03.009914 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:03.009853 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f70232fa-7106-4ec6-8c8e-277c4ce4c719-tls-certs\") pod \"distinct-2-w0-a74d15-kserve-846bdb9948-ht54r\" (UID: \"f70232fa-7106-4ec6-8c8e-277c4ce4c719\") " pod="e2e-models-e2e-worker-w0-a74d15/distinct-2-w0-a74d15-kserve-846bdb9948-ht54r" Oct 30 19:47:03.012619 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:03.012561 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f70232fa-7106-4ec6-8c8e-277c4ce4c719-dshm\") pod \"distinct-2-w0-a74d15-kserve-846bdb9948-ht54r\" (UID: \"f70232fa-7106-4ec6-8c8e-277c4ce4c719\") " pod="e2e-models-e2e-worker-w0-a74d15/distinct-2-w0-a74d15-kserve-846bdb9948-ht54r" Oct 30 19:47:03.070099 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:03.068236 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r248s\" (UniqueName: \"kubernetes.io/projected/f70232fa-7106-4ec6-8c8e-277c4ce4c719-kube-api-access-r248s\") pod \"distinct-2-w0-a74d15-kserve-846bdb9948-ht54r\" (UID: \"f70232fa-7106-4ec6-8c8e-277c4ce4c719\") " pod="e2e-models-e2e-worker-w0-a74d15/distinct-2-w0-a74d15-kserve-846bdb9948-ht54r" Oct 30 19:47:03.113965 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:03.113903 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a74d15/distinct-2-w0-a74d15-kserve-846bdb9948-ht54r" Oct 30 19:47:03.442579 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:03.442492 4292 generic.go:356] "Generic (PLEG): container finished" podID="a179efae-1f42-4b73-b2d7-70f1474c3c9f" containerID="9472ee5955ca5a2b3ac7bfcd49677bc482125e3434d2cb3c61dee8b50e4969cf" exitCode=0 Oct 30 19:47:03.442579 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:03.442549 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-788b6c6b4d-59ggs" Oct 30 19:47:03.442799 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:03.442550 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-788b6c6b4d-59ggs" event={"ID":"a179efae-1f42-4b73-b2d7-70f1474c3c9f","Type":"ContainerDied","Data":"9472ee5955ca5a2b3ac7bfcd49677bc482125e3434d2cb3c61dee8b50e4969cf"} Oct 30 19:47:03.442799 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:03.442680 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-788b6c6b4d-59ggs" event={"ID":"a179efae-1f42-4b73-b2d7-70f1474c3c9f","Type":"ContainerDied","Data":"7623105fb7e3da7709ac11895d4789931ba8a97eadfe5e42f7fe5dbb400b225e"} Oct 30 19:47:03.442799 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:03.442709 4292 scope.go:122] "RemoveContainer" containerID="9472ee5955ca5a2b3ac7bfcd49677bc482125e3434d2cb3c61dee8b50e4969cf" Oct 30 19:47:03.453455 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:03.453426 4292 scope.go:122] "RemoveContainer" containerID="9472ee5955ca5a2b3ac7bfcd49677bc482125e3434d2cb3c61dee8b50e4969cf" Oct 30 19:47:03.453836 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:47:03.453814 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9472ee5955ca5a2b3ac7bfcd49677bc482125e3434d2cb3c61dee8b50e4969cf\": container with ID starting with 9472ee5955ca5a2b3ac7bfcd49677bc482125e3434d2cb3c61dee8b50e4969cf not found: ID does not exist" containerID="9472ee5955ca5a2b3ac7bfcd49677bc482125e3434d2cb3c61dee8b50e4969cf" Oct 30 19:47:03.453940 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:03.453849 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9472ee5955ca5a2b3ac7bfcd49677bc482125e3434d2cb3c61dee8b50e4969cf"} err="failed to get container status \"9472ee5955ca5a2b3ac7bfcd49677bc482125e3434d2cb3c61dee8b50e4969cf\": rpc error: code = NotFound desc = could not find container \"9472ee5955ca5a2b3ac7bfcd49677bc482125e3434d2cb3c61dee8b50e4969cf\": container with ID starting with 9472ee5955ca5a2b3ac7bfcd49677bc482125e3434d2cb3c61dee8b50e4969cf not found: ID does not exist" Oct 30 19:47:03.612985 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:03.612945 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-788b6c6b4d-59ggs"] Oct 30 19:47:03.617029 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:03.617002 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-788b6c6b4d-59ggs"] Oct 30 19:47:03.664759 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:03.664731 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a74d15/distinct-2-w0-a74d15-kserve-846bdb9948-ht54r"] Oct 30 19:47:03.665392 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:47:03.665331 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf70232fa_7106_4ec6_8c8e_277c4ce4c719.slice/crio-7164bae6efcca39366d2cea355c9e7f7183bddf90fbba1c0dc5ba31b1ca2e5af WatchSource:0}: Error finding container 7164bae6efcca39366d2cea355c9e7f7183bddf90fbba1c0dc5ba31b1ca2e5af: Status 404 returned error can't find the container with id 7164bae6efcca39366d2cea355c9e7f7183bddf90fbba1c0dc5ba31b1ca2e5af Oct 30 19:47:04.451001 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:04.450962 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a74d15/distinct-2-w0-a74d15-kserve-846bdb9948-ht54r" event={"ID":"f70232fa-7106-4ec6-8c8e-277c4ce4c719","Type":"ContainerStarted","Data":"1aee1e96ea8dda1200b0f2ac7be9f0d402a0dcfda93a0286dd14f06a84f3d3e3"} Oct 30 19:47:04.451001 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:04.451004 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a74d15/distinct-2-w0-a74d15-kserve-846bdb9948-ht54r" event={"ID":"f70232fa-7106-4ec6-8c8e-277c4ce4c719","Type":"ContainerStarted","Data":"7164bae6efcca39366d2cea355c9e7f7183bddf90fbba1c0dc5ba31b1ca2e5af"} Oct 30 19:47:04.571225 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:04.571171 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-a74d15/distinct-2-w0-a74d15-kserve-846bdb9948-ht54r" podStartSLOduration=2.57115557 podStartE2EDuration="2.57115557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:04.569953487 +0000 UTC m=+1328.088268963" watchObservedRunningTime="2026-10-30 19:47:04.57115557 +0000 UTC m=+1328.089471056" Oct 30 19:47:04.582079 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:04.582043 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a179efae-1f42-4b73-b2d7-70f1474c3c9f" path="/var/lib/kubelet/pods/a179efae-1f42-4b73-b2d7-70f1474c3c9f/volumes" Oct 30 19:47:05.114545 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:05.114509 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a74d15/distinct-2-w0-a74d15-kserve-846bdb9948-ht54r" Oct 30 19:47:13.118455 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:13.118409 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-a74d15/distinct-2-w0-a74d15-kserve-846bdb9948-ht54r" Oct 30 19:47:13.135647 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:13.135615 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-a74d15/distinct-2-w0-a74d15-kserve-846bdb9948-ht54r" Oct 30 19:47:13.266378 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:47:13.261619 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:47:13.742374 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:13.741776 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a74d15/distinct-2-w0-a74d15-kserve-846bdb9948-ht54r" Oct 30 19:47:31.912560 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:31.912512 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp"] Oct 30 19:47:31.913216 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:31.913192 4292 cpu_manager.go:414] "containerMap: removing container" podUID="a179efae-1f42-4b73-b2d7-70f1474c3c9f" containerName="payload-processing" Oct 30 19:47:31.913329 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:31.913218 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a179efae-1f42-4b73-b2d7-70f1474c3c9f" containerName="payload-processing" Oct 30 19:47:31.913420 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:31.913407 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="a179efae-1f42-4b73-b2d7-70f1474c3c9f" containerName="payload-processing" Oct 30 19:47:31.917190 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:31.917165 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp" Oct 30 19:47:31.976755 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:31.976721 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-a74d15-openshift-default-dockercfg-pfbt9\"" Oct 30 19:47:32.054374 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:32.048938 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp"] Oct 30 19:47:32.062702 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:32.062667 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb-istiod-ca-cert\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp\" (UID: \"34aa822f-7a67-4e27-9acb-6e8b0b80a5cb\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp" Oct 30 19:47:32.062947 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:32.062913 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb-istio-envoy\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp\" (UID: \"34aa822f-7a67-4e27-9acb-6e8b0b80a5cb\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp" Oct 30 19:47:32.063029 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:32.062952 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb-istio-data\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp\" (UID: \"34aa822f-7a67-4e27-9acb-6e8b0b80a5cb\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp" Oct 30 19:47:32.063029 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:32.062995 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb-workload-certs\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp\" (UID: \"34aa822f-7a67-4e27-9acb-6e8b0b80a5cb\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp" Oct 30 19:47:32.063140 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:32.063028 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf5vx\" (UniqueName: \"kubernetes.io/projected/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb-kube-api-access-gf5vx\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp\" (UID: \"34aa822f-7a67-4e27-9acb-6e8b0b80a5cb\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp" Oct 30 19:47:32.063140 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:32.063127 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb-credential-socket\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp\" (UID: \"34aa822f-7a67-4e27-9acb-6e8b0b80a5cb\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp" Oct 30 19:47:32.063242 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:32.063171 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb-istio-podinfo\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp\" (UID: \"34aa822f-7a67-4e27-9acb-6e8b0b80a5cb\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp" Oct 30 19:47:32.063295 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:32.063240 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb-istio-token\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp\" (UID: \"34aa822f-7a67-4e27-9acb-6e8b0b80a5cb\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp" Oct 30 19:47:32.063295 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:32.063282 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb-workload-socket\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp\" (UID: \"34aa822f-7a67-4e27-9acb-6e8b0b80a5cb\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp" Oct 30 19:47:32.164478 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:32.164402 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb-istiod-ca-cert\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp\" (UID: \"34aa822f-7a67-4e27-9acb-6e8b0b80a5cb\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp" Oct 30 19:47:32.164741 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:32.164721 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb-istio-envoy\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp\" (UID: \"34aa822f-7a67-4e27-9acb-6e8b0b80a5cb\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp" Oct 30 19:47:32.164869 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:32.164853 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb-istio-data\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp\" (UID: \"34aa822f-7a67-4e27-9acb-6e8b0b80a5cb\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp" Oct 30 19:47:32.164995 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:32.164980 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb-workload-certs\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp\" (UID: \"34aa822f-7a67-4e27-9acb-6e8b0b80a5cb\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp" Oct 30 19:47:32.165113 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:32.165098 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf5vx\" (UniqueName: \"kubernetes.io/projected/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb-kube-api-access-gf5vx\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp\" (UID: \"34aa822f-7a67-4e27-9acb-6e8b0b80a5cb\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp" Oct 30 19:47:32.165310 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:32.165291 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb-credential-socket\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp\" (UID: \"34aa822f-7a67-4e27-9acb-6e8b0b80a5cb\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp" Oct 30 19:47:32.165474 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:32.165458 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb-istio-podinfo\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp\" (UID: \"34aa822f-7a67-4e27-9acb-6e8b0b80a5cb\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp" Oct 30 19:47:32.165636 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:32.165619 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb-istio-token\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp\" (UID: \"34aa822f-7a67-4e27-9acb-6e8b0b80a5cb\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp" Oct 30 19:47:32.165760 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:32.165745 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb-workload-socket\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp\" (UID: \"34aa822f-7a67-4e27-9acb-6e8b0b80a5cb\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp" Oct 30 19:47:32.166118 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:32.166102 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb-workload-socket\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp\" (UID: \"34aa822f-7a67-4e27-9acb-6e8b0b80a5cb\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp" Oct 30 19:47:32.166424 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:32.166409 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb-workload-certs\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp\" (UID: \"34aa822f-7a67-4e27-9acb-6e8b0b80a5cb\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp" Oct 30 19:47:32.166961 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:32.165690 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb-istio-data\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp\" (UID: \"34aa822f-7a67-4e27-9acb-6e8b0b80a5cb\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp" Oct 30 19:47:32.167264 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:32.167250 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb-credential-socket\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp\" (UID: \"34aa822f-7a67-4e27-9acb-6e8b0b80a5cb\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp" Oct 30 19:47:32.167372 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:32.165341 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb-istiod-ca-cert\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp\" (UID: \"34aa822f-7a67-4e27-9acb-6e8b0b80a5cb\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp" Oct 30 19:47:32.172802 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:32.172780 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb-istio-envoy\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp\" (UID: \"34aa822f-7a67-4e27-9acb-6e8b0b80a5cb\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp" Oct 30 19:47:32.178421 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:32.178398 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb-istio-podinfo\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp\" (UID: \"34aa822f-7a67-4e27-9acb-6e8b0b80a5cb\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp" Oct 30 19:47:32.229955 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:32.229920 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf5vx\" (UniqueName: \"kubernetes.io/projected/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb-kube-api-access-gf5vx\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp\" (UID: \"34aa822f-7a67-4e27-9acb-6e8b0b80a5cb\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp" Oct 30 19:47:32.230970 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:32.230944 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb-istio-token\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp\" (UID: \"34aa822f-7a67-4e27-9acb-6e8b0b80a5cb\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp" Oct 30 19:47:32.243139 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:32.242888 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp" Oct 30 19:47:32.938379 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:32.936342 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp"] Oct 30 19:47:33.017545 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:47:33.015649 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34aa822f_7a67_4e27_9acb_6e8b0b80a5cb.slice/crio-a28a391acc91330fd35267147fe8d82f337a43d59735e119c957b27045e85b7e WatchSource:0}: Error finding container a28a391acc91330fd35267147fe8d82f337a43d59735e119c957b27045e85b7e: Status 404 returned error can't find the container with id a28a391acc91330fd35267147fe8d82f337a43d59735e119c957b27045e85b7e Oct 30 19:47:33.029376 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:33.026273 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:47:33.029376 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:33.026397 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:47:33.029376 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:33.026449 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:47:33.169380 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:33.165119 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml"] Oct 30 19:47:33.197457 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:33.195548 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml" Oct 30 19:47:33.254375 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:33.247223 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml"] Oct 30 19:47:33.396617 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:33.393232 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69d11c12-c4cb-452e-90cc-4cbe03333951-istio-envoy\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml\" (UID: \"69d11c12-c4cb-452e-90cc-4cbe03333951\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml" Oct 30 19:47:33.396617 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:33.393310 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69d11c12-c4cb-452e-90cc-4cbe03333951-workload-certs\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml\" (UID: \"69d11c12-c4cb-452e-90cc-4cbe03333951\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml" Oct 30 19:47:33.396617 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:33.393396 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69d11c12-c4cb-452e-90cc-4cbe03333951-istio-podinfo\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml\" (UID: \"69d11c12-c4cb-452e-90cc-4cbe03333951\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml" Oct 30 19:47:33.396617 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:33.393428 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69d11c12-c4cb-452e-90cc-4cbe03333951-istiod-ca-cert\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml\" (UID: \"69d11c12-c4cb-452e-90cc-4cbe03333951\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml" Oct 30 19:47:33.396617 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:33.393464 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69d11c12-c4cb-452e-90cc-4cbe03333951-credential-socket\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml\" (UID: \"69d11c12-c4cb-452e-90cc-4cbe03333951\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml" Oct 30 19:47:33.396617 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:33.393500 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69d11c12-c4cb-452e-90cc-4cbe03333951-workload-socket\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml\" (UID: \"69d11c12-c4cb-452e-90cc-4cbe03333951\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml" Oct 30 19:47:33.396617 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:33.393542 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2nz5\" (UniqueName: \"kubernetes.io/projected/69d11c12-c4cb-452e-90cc-4cbe03333951-kube-api-access-x2nz5\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml\" (UID: \"69d11c12-c4cb-452e-90cc-4cbe03333951\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml" Oct 30 19:47:33.396617 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:33.393617 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69d11c12-c4cb-452e-90cc-4cbe03333951-istio-token\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml\" (UID: \"69d11c12-c4cb-452e-90cc-4cbe03333951\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml" Oct 30 19:47:33.422377 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:33.418499 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69d11c12-c4cb-452e-90cc-4cbe03333951-istio-data\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml\" (UID: \"69d11c12-c4cb-452e-90cc-4cbe03333951\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml" Oct 30 19:47:33.584837 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:33.584798 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69d11c12-c4cb-452e-90cc-4cbe03333951-istio-data\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml\" (UID: \"69d11c12-c4cb-452e-90cc-4cbe03333951\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml" Oct 30 19:47:33.585227 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:33.585192 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69d11c12-c4cb-452e-90cc-4cbe03333951-istio-envoy\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml\" (UID: \"69d11c12-c4cb-452e-90cc-4cbe03333951\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml" Oct 30 19:47:33.585459 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:33.585438 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69d11c12-c4cb-452e-90cc-4cbe03333951-workload-certs\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml\" (UID: \"69d11c12-c4cb-452e-90cc-4cbe03333951\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml" Oct 30 19:47:33.585689 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:33.585670 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69d11c12-c4cb-452e-90cc-4cbe03333951-istio-podinfo\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml\" (UID: \"69d11c12-c4cb-452e-90cc-4cbe03333951\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml" Oct 30 19:47:33.628518 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:33.612034 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69d11c12-c4cb-452e-90cc-4cbe03333951-istio-data\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml\" (UID: \"69d11c12-c4cb-452e-90cc-4cbe03333951\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml" Oct 30 19:47:33.628518 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:33.613298 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69d11c12-c4cb-452e-90cc-4cbe03333951-istiod-ca-cert\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml\" (UID: \"69d11c12-c4cb-452e-90cc-4cbe03333951\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml" Oct 30 19:47:33.628518 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:33.612446 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69d11c12-c4cb-452e-90cc-4cbe03333951-istiod-ca-cert\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml\" (UID: \"69d11c12-c4cb-452e-90cc-4cbe03333951\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml" Oct 30 19:47:33.628518 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:33.613452 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69d11c12-c4cb-452e-90cc-4cbe03333951-credential-socket\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml\" (UID: \"69d11c12-c4cb-452e-90cc-4cbe03333951\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml" Oct 30 19:47:33.628518 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:33.613495 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69d11c12-c4cb-452e-90cc-4cbe03333951-workload-socket\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml\" (UID: \"69d11c12-c4cb-452e-90cc-4cbe03333951\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml" Oct 30 19:47:33.628518 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:33.613542 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2nz5\" (UniqueName: \"kubernetes.io/projected/69d11c12-c4cb-452e-90cc-4cbe03333951-kube-api-access-x2nz5\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml\" (UID: \"69d11c12-c4cb-452e-90cc-4cbe03333951\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml" Oct 30 19:47:33.628518 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:33.613626 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69d11c12-c4cb-452e-90cc-4cbe03333951-istio-token\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml\" (UID: \"69d11c12-c4cb-452e-90cc-4cbe03333951\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml" Oct 30 19:47:33.628518 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:33.618039 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69d11c12-c4cb-452e-90cc-4cbe03333951-istio-envoy\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml\" (UID: \"69d11c12-c4cb-452e-90cc-4cbe03333951\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml" Oct 30 19:47:33.628518 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:33.618335 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69d11c12-c4cb-452e-90cc-4cbe03333951-workload-certs\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml\" (UID: \"69d11c12-c4cb-452e-90cc-4cbe03333951\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml" Oct 30 19:47:33.680888 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:33.629706 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69d11c12-c4cb-452e-90cc-4cbe03333951-credential-socket\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml\" (UID: \"69d11c12-c4cb-452e-90cc-4cbe03333951\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml" Oct 30 19:47:33.680888 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:33.630001 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69d11c12-c4cb-452e-90cc-4cbe03333951-workload-socket\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml\" (UID: \"69d11c12-c4cb-452e-90cc-4cbe03333951\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml" Oct 30 19:47:33.680888 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:33.630268 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ec609/premium-w4-4ec609-kserve-69f7c88bf7-rzmmt" podUID="fb6be431-0a6c-4d81-b365-d7537f5d385a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 19:47:33.680888 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:33.657890 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-98a19b/simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r" podUID="1f35e32e-1f09-413d-a364-14b8ac1019f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:47:33.692090 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:33.691998 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69d11c12-c4cb-452e-90cc-4cbe03333951-istio-podinfo\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml\" (UID: \"69d11c12-c4cb-452e-90cc-4cbe03333951\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml" Oct 30 19:47:33.723445 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:33.722985 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69d11c12-c4cb-452e-90cc-4cbe03333951-istio-token\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml\" (UID: \"69d11c12-c4cb-452e-90cc-4cbe03333951\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml" Oct 30 19:47:33.726604 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:33.726499 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp" event={"ID":"34aa822f-7a67-4e27-9acb-6e8b0b80a5cb","Type":"ContainerStarted","Data":"a28a391acc91330fd35267147fe8d82f337a43d59735e119c957b27045e85b7e"} Oct 30 19:47:33.748808 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:33.748711 4292 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-bbfa07/distinct-w3-bbfa07-kserve-67bb47494b-t8t7l" podUID="16def41a-a612-4a4c-ada9-bb170b066af2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:47:33.787345 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:33.787253 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2nz5\" (UniqueName: \"kubernetes.io/projected/69d11c12-c4cb-452e-90cc-4cbe03333951-kube-api-access-x2nz5\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml\" (UID: \"69d11c12-c4cb-452e-90cc-4cbe03333951\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml" Oct 30 19:47:33.804318 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:33.803464 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bbfa07/premium-w3-bbfa07-kserve-5675444c9-x42d7" podUID="e0b75746-c06c-407d-8f61-c795c519aad6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:47:33.830172 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:33.829878 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml" Oct 30 19:47:34.635686 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:34.635542 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml"] Oct 30 19:47:34.644675 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:34.644439 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:47:34.644675 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:34.644529 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:47:34.644675 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:34.644573 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:47:34.734547 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:34.734509 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp" event={"ID":"34aa822f-7a67-4e27-9acb-6e8b0b80a5cb","Type":"ContainerStarted","Data":"1dd48400431929c6a72fb963b6a448ac7c4fb9f534f394f4af7231e53f2a8e8c"} Oct 30 19:47:34.736289 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:34.736257 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml" event={"ID":"69d11c12-c4cb-452e-90cc-4cbe03333951","Type":"ContainerStarted","Data":"892028e27c2c1381e5591f3137d7fa9e42246b8c1765844d2125be6c46c37521"} Oct 30 19:47:34.866379 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:34.865852 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp" podStartSLOduration=3.865830483 podStartE2EDuration="3.865830483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:34.865723485 +0000 UTC m=+1358.384038979" watchObservedRunningTime="2026-10-30 19:47:34.865830483 +0000 UTC m=+1358.384145941" Oct 30 19:47:35.248334 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:35.248286 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp" Oct 30 19:47:35.276479 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:35.272510 4292 patch_prober.go:29] interesting pod/e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 19:47:35.276479 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:35.272602 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp" podUID="34aa822f-7a67-4e27-9acb-6e8b0b80a5cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 19:47:35.746834 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:35.746784 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml" event={"ID":"69d11c12-c4cb-452e-90cc-4cbe03333951","Type":"ContainerStarted","Data":"90bddd7abe3fb567b216e0ce5a0a6e4226b0e4e16c1fcf8e001744bd1ed7b18b"} Oct 30 19:47:35.830989 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:35.830954 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml" Oct 30 19:47:35.831803 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:35.831776 4292 patch_prober.go:29] interesting pod/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 19:47:35.831937 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:35.831850 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml" podUID="69d11c12-c4cb-452e-90cc-4cbe03333951" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 19:47:35.972377 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:35.969467 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml" podStartSLOduration=2.969442844 podStartE2EDuration="2.969442844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:35.936376772 +0000 UTC m=+1359.454692230" watchObservedRunningTime="2026-10-30 19:47:35.969442844 +0000 UTC m=+1359.487758302" Oct 30 19:47:36.244036 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:36.243997 4292 patch_prober.go:29] interesting pod/e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 19:47:36.244237 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:36.244062 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp" podUID="34aa822f-7a67-4e27-9acb-6e8b0b80a5cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 19:47:36.831293 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:36.831259 4292 patch_prober.go:29] interesting pod/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 19:47:36.831832 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:36.831318 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml" podUID="69d11c12-c4cb-452e-90cc-4cbe03333951" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 19:47:37.244507 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:37.244423 4292 patch_prober.go:29] interesting pod/e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 19:47:37.244507 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:37.244481 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp" podUID="34aa822f-7a67-4e27-9acb-6e8b0b80a5cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 19:47:37.831806 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:37.831770 4292 patch_prober.go:29] interesting pod/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 19:47:37.832210 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:37.831856 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml" podUID="69d11c12-c4cb-452e-90cc-4cbe03333951" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 19:47:38.244616 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:38.244532 4292 patch_prober.go:29] interesting pod/e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 19:47:38.244616 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:38.244590 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp" podUID="34aa822f-7a67-4e27-9acb-6e8b0b80a5cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 19:47:38.831492 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:38.831455 4292 patch_prober.go:29] interesting pod/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 19:47:38.831678 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:38.831520 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml" podUID="69d11c12-c4cb-452e-90cc-4cbe03333951" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 19:47:39.243922 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:39.243830 4292 patch_prober.go:29] interesting pod/e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 19:47:39.243922 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:39.243896 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp" podUID="34aa822f-7a67-4e27-9acb-6e8b0b80a5cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 19:47:39.831689 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:39.831653 4292 patch_prober.go:29] interesting pod/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 19:47:39.831865 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:39.831719 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml" podUID="69d11c12-c4cb-452e-90cc-4cbe03333951" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 19:47:40.244611 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:40.244531 4292 patch_prober.go:29] interesting pod/e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 19:47:40.245012 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:40.244604 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp" podUID="34aa822f-7a67-4e27-9acb-6e8b0b80a5cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 19:47:40.831196 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:40.831154 4292 patch_prober.go:29] interesting pod/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 19:47:40.831393 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:40.831223 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml" podUID="69d11c12-c4cb-452e-90cc-4cbe03333951" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 19:47:41.244600 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:41.244510 4292 patch_prober.go:29] interesting pod/e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 19:47:41.244600 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:41.244563 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp" podUID="34aa822f-7a67-4e27-9acb-6e8b0b80a5cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 19:47:41.832411 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:41.830980 4292 patch_prober.go:29] interesting pod/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 19:47:41.832411 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:41.831046 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml" podUID="69d11c12-c4cb-452e-90cc-4cbe03333951" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 19:47:42.249180 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:42.249088 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp" Oct 30 19:47:42.249666 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:42.249519 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp" Oct 30 19:47:42.250141 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:42.250119 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp" Oct 30 19:47:42.835091 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:42.835057 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml" Oct 30 19:47:42.835388 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:42.835346 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml" Oct 30 19:47:42.836140 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:42.836119 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml" Oct 30 19:47:43.227265 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:47:43.227186 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:47:57.336212 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:57.336172 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b95df7bc-qddlf"] Oct 30 19:47:57.350383 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:57.347222 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95df7bc-qddlf" Oct 30 19:47:57.369626 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:57.369591 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07c36c0a-58d5-4031-855c-bf07dfd047bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b95df7bc-qddlf\" (UID: \"07c36c0a-58d5-4031-855c-bf07dfd047bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95df7bc-qddlf" Oct 30 19:47:57.369891 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:57.369871 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vf7t\" (UniqueName: \"kubernetes.io/projected/07c36c0a-58d5-4031-855c-bf07dfd047bc-kube-api-access-9vf7t\") pod \"maas-api-revoke-sub-b95df7bc-qddlf\" (UID: \"07c36c0a-58d5-4031-855c-bf07dfd047bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95df7bc-qddlf" Oct 30 19:47:57.370122 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:57.370104 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07c36c0a-58d5-4031-855c-bf07dfd047bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b95df7bc-qddlf\" (UID: \"07c36c0a-58d5-4031-855c-bf07dfd047bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95df7bc-qddlf" Oct 30 19:47:57.371782 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:57.371759 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b95df7bc-qddlf"] Oct 30 19:47:57.416977 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:57.416954 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:47:57.417109 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:57.417052 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\"" Oct 30 19:47:57.471821 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:57.471790 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vf7t\" (UniqueName: \"kubernetes.io/projected/07c36c0a-58d5-4031-855c-bf07dfd047bc-kube-api-access-9vf7t\") pod \"maas-api-revoke-sub-b95df7bc-qddlf\" (UID: \"07c36c0a-58d5-4031-855c-bf07dfd047bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95df7bc-qddlf" Oct 30 19:47:57.471976 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:57.471848 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07c36c0a-58d5-4031-855c-bf07dfd047bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b95df7bc-qddlf\" (UID: \"07c36c0a-58d5-4031-855c-bf07dfd047bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95df7bc-qddlf" Oct 30 19:47:57.471976 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:57.471942 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07c36c0a-58d5-4031-855c-bf07dfd047bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b95df7bc-qddlf\" (UID: \"07c36c0a-58d5-4031-855c-bf07dfd047bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95df7bc-qddlf" Oct 30 19:47:57.472525 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:57.472493 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07c36c0a-58d5-4031-855c-bf07dfd047bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b95df7bc-qddlf\" (UID: \"07c36c0a-58d5-4031-855c-bf07dfd047bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95df7bc-qddlf" Oct 30 19:47:57.474763 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:57.474742 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07c36c0a-58d5-4031-855c-bf07dfd047bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b95df7bc-qddlf\" (UID: \"07c36c0a-58d5-4031-855c-bf07dfd047bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95df7bc-qddlf" Oct 30 19:47:57.534725 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:57.534696 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vf7t\" (UniqueName: \"kubernetes.io/projected/07c36c0a-58d5-4031-855c-bf07dfd047bc-kube-api-access-9vf7t\") pod \"maas-api-revoke-sub-b95df7bc-qddlf\" (UID: \"07c36c0a-58d5-4031-855c-bf07dfd047bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95df7bc-qddlf" Oct 30 19:47:57.670052 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:57.669977 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95df7bc-qddlf" Oct 30 19:47:58.046159 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:58.046130 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b95df7bc-qddlf"] Oct 30 19:47:58.046909 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:47:58.046873 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07c36c0a_58d5_4031_855c_bf07dfd047bc.slice/crio-3d945f2eff25da4853d32d70d312168130894873e8e7da497d07512c5f8a5a9e WatchSource:0}: Error finding container 3d945f2eff25da4853d32d70d312168130894873e8e7da497d07512c5f8a5a9e: Status 404 returned error can't find the container with id 3d945f2eff25da4853d32d70d312168130894873e8e7da497d07512c5f8a5a9e Oct 30 19:47:58.993132 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:58.993096 4292 generic.go:356] "Generic (PLEG): container finished" podID="07c36c0a-58d5-4031-855c-bf07dfd047bc" containerID="82f6512e75490aa5f6e24e5aa087b12a77d3c56a6f2af9d67d7d9bebbd17fb05" exitCode=0 Oct 30 19:47:58.993546 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:58.993156 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95df7bc-qddlf" event={"ID":"07c36c0a-58d5-4031-855c-bf07dfd047bc","Type":"ContainerDied","Data":"82f6512e75490aa5f6e24e5aa087b12a77d3c56a6f2af9d67d7d9bebbd17fb05"} Oct 30 19:47:58.993546 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:47:58.993211 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95df7bc-qddlf" event={"ID":"07c36c0a-58d5-4031-855c-bf07dfd047bc","Type":"ContainerStarted","Data":"3d945f2eff25da4853d32d70d312168130894873e8e7da497d07512c5f8a5a9e"} Oct 30 19:48:00.145564 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:00.145538 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95df7bc-qddlf" Oct 30 19:48:00.200560 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:00.200534 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07c36c0a-58d5-4031-855c-bf07dfd047bc-kube-api-access-9vf7t\" (UniqueName: \"kubernetes.io/projected/07c36c0a-58d5-4031-855c-bf07dfd047bc-kube-api-access-9vf7t\") pod \"07c36c0a-58d5-4031-855c-bf07dfd047bc\" (UID: \"07c36c0a-58d5-4031-855c-bf07dfd047bc\") " Oct 30 19:48:00.200709 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:00.200622 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07c36c0a-58d5-4031-855c-bf07dfd047bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07c36c0a-58d5-4031-855c-bf07dfd047bc-maas-api-openshift-ca\") pod \"07c36c0a-58d5-4031-855c-bf07dfd047bc\" (UID: \"07c36c0a-58d5-4031-855c-bf07dfd047bc\") " Oct 30 19:48:00.200709 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:00.200664 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07c36c0a-58d5-4031-855c-bf07dfd047bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07c36c0a-58d5-4031-855c-bf07dfd047bc-maas-api-xks-ca\") pod \"07c36c0a-58d5-4031-855c-bf07dfd047bc\" (UID: \"07c36c0a-58d5-4031-855c-bf07dfd047bc\") " Oct 30 19:48:00.200994 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:00.200972 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07c36c0a-58d5-4031-855c-bf07dfd047bc-maas-api-openshift-ca" pod "07c36c0a-58d5-4031-855c-bf07dfd047bc" (UID: "07c36c0a-58d5-4031-855c-bf07dfd047bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:48:00.202576 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:00.202554 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07c36c0a-58d5-4031-855c-bf07dfd047bc-kube-api-access-9vf7t" pod "07c36c0a-58d5-4031-855c-bf07dfd047bc" (UID: "07c36c0a-58d5-4031-855c-bf07dfd047bc"). InnerVolumeSpecName "kube-api-access-9vf7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:48:00.202576 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:00.202564 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07c36c0a-58d5-4031-855c-bf07dfd047bc-maas-api-xks-ca" pod "07c36c0a-58d5-4031-855c-bf07dfd047bc" (UID: "07c36c0a-58d5-4031-855c-bf07dfd047bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:48:00.302288 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:00.302258 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vf7t\" (UniqueName: \"kubernetes.io/projected/07c36c0a-58d5-4031-855c-bf07dfd047bc-kube-api-access-9vf7t\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:48:00.302288 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:00.302284 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07c36c0a-58d5-4031-855c-bf07dfd047bc-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:48:00.302288 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:00.302294 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07c36c0a-58d5-4031-855c-bf07dfd047bc-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:48:01.005118 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:01.005035 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95df7bc-qddlf" event={"ID":"07c36c0a-58d5-4031-855c-bf07dfd047bc","Type":"ContainerDied","Data":"3d945f2eff25da4853d32d70d312168130894873e8e7da497d07512c5f8a5a9e"} Oct 30 19:48:01.005118 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:01.005071 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95df7bc-qddlf" Oct 30 19:48:01.005118 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:01.005084 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d945f2eff25da4853d32d70d312168130894873e8e7da497d07512c5f8a5a9e" Oct 30 19:48:13.221957 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:48:13.221871 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:48:19.610378 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:19.609463 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ec609/premium-w4-4ec609-kserve-69f7c88bf7-rzmmt" podUID="fb6be431-0a6c-4d81-b365-d7537f5d385a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:48:19.656497 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:19.656441 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-98a19b/simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r" podUID="1f35e32e-1f09-413d-a364-14b8ac1019f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:48:19.684506 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:19.684446 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bbfa07/distinct-w3-bbfa07-kserve-67bb47494b-t8t7l" podUID="16def41a-a612-4a4c-ada9-bb170b066af2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:48:19.799664 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:19.798448 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bbfa07/premium-w3-bbfa07-kserve-5675444c9-x42d7" podUID="e0b75746-c06c-407d-8f61-c795c519aad6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:48:20.186088 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:20.185438 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bbfa07/distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2" podUID="f562f81a-ebd3-4efa-889d-bf4a757527fe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:48:20.469433 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:20.468448 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a74d15/premium-w0-a74d15-kserve-864f45fdfb-949gm" podUID="eb0ced80-4c4f-41dc-883c-b1a21e0f11f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:48:20.660388 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:20.641605 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ec609/premium-w4-4ec609-kserve-69f7c88bf7-rzmmt" podUID="fb6be431-0a6c-4d81-b365-d7537f5d385a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:48:20.777435 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:20.764372 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-98a19b/simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r" podUID="1f35e32e-1f09-413d-a364-14b8ac1019f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 19:48:20.777435 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:20.764769 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bbfa07/distinct-w3-bbfa07-kserve-67bb47494b-t8t7l" podUID="16def41a-a612-4a4c-ada9-bb170b066af2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 30 19:48:20.874377 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:20.831263 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bbfa07/premium-w3-bbfa07-kserve-5675444c9-x42d7" podUID="e0b75746-c06c-407d-8f61-c795c519aad6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 19:48:20.973376 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:20.972920 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a74d15/distinct-2-w0-a74d15-kserve-846bdb9948-ht54r" podUID="f70232fa-7106-4ec6-8c8e-277c4ce4c719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:48:21.192435 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:21.190024 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bbfa07/distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2" podUID="f562f81a-ebd3-4efa-889d-bf4a757527fe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:48:21.427378 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:21.308729 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ec609/premium-w4-4ec609-kserve-69f7c88bf7-rzmmt" Oct 30 19:48:21.453625 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:21.453539 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-98a19b/simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r" Oct 30 19:48:21.457374 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:21.453876 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bbfa07/distinct-w3-bbfa07-kserve-67bb47494b-t8t7l" Oct 30 19:48:21.469376 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:21.465457 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bbfa07/premium-w3-bbfa07-kserve-5675444c9-x42d7" Oct 30 19:48:21.489900 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:21.489864 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-98a19b/simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r" Oct 30 19:48:21.519486 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:21.519452 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bbfa07/distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2" Oct 30 19:48:21.520238 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:21.518326 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a74d15/premium-w0-a74d15-kserve-864f45fdfb-949gm" podUID="eb0ced80-4c4f-41dc-883c-b1a21e0f11f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:48:21.520882 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:21.520861 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a74d15/premium-w0-a74d15-kserve-864f45fdfb-949gm" Oct 30 19:48:21.696155 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:21.696101 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ec609/premium-w4-4ec609-kserve-69f7c88bf7-rzmmt" podUID="fb6be431-0a6c-4d81-b365-d7537f5d385a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 19:48:21.770016 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:21.769633 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bbfa07/distinct-w3-bbfa07-kserve-67bb47494b-t8t7l" podUID="16def41a-a612-4a4c-ada9-bb170b066af2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:48:21.875377 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:21.857442 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bbfa07/premium-w3-bbfa07-kserve-5675444c9-x42d7" podUID="e0b75746-c06c-407d-8f61-c795c519aad6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:48:21.984492 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:21.983619 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a74d15/distinct-2-w0-a74d15-kserve-846bdb9948-ht54r" podUID="f70232fa-7106-4ec6-8c8e-277c4ce4c719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:48:21.985196 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:21.985157 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a74d15/distinct-2-w0-a74d15-kserve-846bdb9948-ht54r" Oct 30 19:48:22.351378 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:22.344390 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bbfa07/distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2" podUID="f562f81a-ebd3-4efa-889d-bf4a757527fe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 30 19:48:22.405040 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:22.405010 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a74d15/premium-w0-a74d15-kserve-864f45fdfb-949gm" Oct 30 19:48:22.565668 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:22.565641 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ec609/premium-w4-4ec609-kserve-69f7c88bf7-rzmmt" Oct 30 19:48:22.658461 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:22.649822 4292 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-bbfa07/premium-w3-bbfa07-kserve-5675444c9-x42d7" podUID="e0b75746-c06c-407d-8f61-c795c519aad6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:48:22.691282 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:22.691251 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bbfa07/distinct-w3-bbfa07-kserve-67bb47494b-t8t7l" Oct 30 19:48:22.743184 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:22.742989 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bbfa07/distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2" Oct 30 19:48:22.745395 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:22.745210 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bbfa07/premium-w3-bbfa07-kserve-5675444c9-x42d7" Oct 30 19:48:22.763690 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:22.763665 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a74d15/distinct-2-w0-a74d15-kserve-846bdb9948-ht54r" Oct 30 19:48:36.137054 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:36.137019 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2"] Oct 30 19:48:36.137625 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:36.137607 4292 cpu_manager.go:414] "containerMap: removing container" podUID="07c36c0a-58d5-4031-855c-bf07dfd047bc" containerName="revoke-keys" Oct 30 19:48:36.137694 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:36.137626 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07c36c0a-58d5-4031-855c-bf07dfd047bc" containerName="revoke-keys" Oct 30 19:48:36.137762 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:36.137747 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="07c36c0a-58d5-4031-855c-bf07dfd047bc" containerName="revoke-keys" Oct 30 19:48:36.141519 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:36.141493 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2" Oct 30 19:48:36.188500 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:36.188467 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef78c2a5-79c3-45af-aad6-26743703d25e-istio-envoy\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2\" (UID: \"ef78c2a5-79c3-45af-aad6-26743703d25e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2" Oct 30 19:48:36.188500 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:36.188500 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef78c2a5-79c3-45af-aad6-26743703d25e-istiod-ca-cert\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2\" (UID: \"ef78c2a5-79c3-45af-aad6-26743703d25e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2" Oct 30 19:48:36.188682 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:36.188574 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef78c2a5-79c3-45af-aad6-26743703d25e-istio-podinfo\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2\" (UID: \"ef78c2a5-79c3-45af-aad6-26743703d25e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2" Oct 30 19:48:36.188682 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:36.188601 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef78c2a5-79c3-45af-aad6-26743703d25e-workload-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2\" (UID: \"ef78c2a5-79c3-45af-aad6-26743703d25e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2" Oct 30 19:48:36.188747 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:36.188712 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef78c2a5-79c3-45af-aad6-26743703d25e-istio-token\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2\" (UID: \"ef78c2a5-79c3-45af-aad6-26743703d25e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2" Oct 30 19:48:36.188782 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:36.188758 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxz7f\" (UniqueName: \"kubernetes.io/projected/ef78c2a5-79c3-45af-aad6-26743703d25e-kube-api-access-fxz7f\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2\" (UID: \"ef78c2a5-79c3-45af-aad6-26743703d25e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2" Oct 30 19:48:36.188865 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:36.188844 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef78c2a5-79c3-45af-aad6-26743703d25e-credential-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2\" (UID: \"ef78c2a5-79c3-45af-aad6-26743703d25e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2" Oct 30 19:48:36.188902 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:36.188878 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef78c2a5-79c3-45af-aad6-26743703d25e-istio-data\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2\" (UID: \"ef78c2a5-79c3-45af-aad6-26743703d25e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2" Oct 30 19:48:36.188934 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:36.188907 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef78c2a5-79c3-45af-aad6-26743703d25e-workload-certs\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2\" (UID: \"ef78c2a5-79c3-45af-aad6-26743703d25e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2" Oct 30 19:48:36.199078 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:36.199046 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2"] Oct 30 19:48:36.290376 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:36.290312 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef78c2a5-79c3-45af-aad6-26743703d25e-istio-envoy\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2\" (UID: \"ef78c2a5-79c3-45af-aad6-26743703d25e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2" Oct 30 19:48:36.290588 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:36.290380 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef78c2a5-79c3-45af-aad6-26743703d25e-istiod-ca-cert\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2\" (UID: \"ef78c2a5-79c3-45af-aad6-26743703d25e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2" Oct 30 19:48:36.290588 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:36.290456 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef78c2a5-79c3-45af-aad6-26743703d25e-istio-podinfo\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2\" (UID: \"ef78c2a5-79c3-45af-aad6-26743703d25e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2" Oct 30 19:48:36.290588 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:36.290502 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef78c2a5-79c3-45af-aad6-26743703d25e-workload-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2\" (UID: \"ef78c2a5-79c3-45af-aad6-26743703d25e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2" Oct 30 19:48:36.290588 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:36.290561 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef78c2a5-79c3-45af-aad6-26743703d25e-istio-token\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2\" (UID: \"ef78c2a5-79c3-45af-aad6-26743703d25e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2" Oct 30 19:48:36.290803 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:36.290613 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxz7f\" (UniqueName: \"kubernetes.io/projected/ef78c2a5-79c3-45af-aad6-26743703d25e-kube-api-access-fxz7f\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2\" (UID: \"ef78c2a5-79c3-45af-aad6-26743703d25e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2" Oct 30 19:48:36.290889 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:36.290824 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef78c2a5-79c3-45af-aad6-26743703d25e-credential-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2\" (UID: \"ef78c2a5-79c3-45af-aad6-26743703d25e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2" Oct 30 19:48:36.290957 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:36.290908 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef78c2a5-79c3-45af-aad6-26743703d25e-istio-data\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2\" (UID: \"ef78c2a5-79c3-45af-aad6-26743703d25e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2" Oct 30 19:48:36.290957 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:36.290918 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef78c2a5-79c3-45af-aad6-26743703d25e-workload-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2\" (UID: \"ef78c2a5-79c3-45af-aad6-26743703d25e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2" Oct 30 19:48:36.290957 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:36.290947 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef78c2a5-79c3-45af-aad6-26743703d25e-workload-certs\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2\" (UID: \"ef78c2a5-79c3-45af-aad6-26743703d25e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2" Oct 30 19:48:36.291153 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:36.291126 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef78c2a5-79c3-45af-aad6-26743703d25e-credential-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2\" (UID: \"ef78c2a5-79c3-45af-aad6-26743703d25e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2" Oct 30 19:48:36.291222 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:36.291207 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef78c2a5-79c3-45af-aad6-26743703d25e-istio-data\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2\" (UID: \"ef78c2a5-79c3-45af-aad6-26743703d25e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2" Oct 30 19:48:36.291324 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:36.291298 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef78c2a5-79c3-45af-aad6-26743703d25e-istiod-ca-cert\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2\" (UID: \"ef78c2a5-79c3-45af-aad6-26743703d25e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2" Oct 30 19:48:36.291417 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:36.291313 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef78c2a5-79c3-45af-aad6-26743703d25e-workload-certs\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2\" (UID: \"ef78c2a5-79c3-45af-aad6-26743703d25e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2" Oct 30 19:48:36.292739 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:36.292714 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef78c2a5-79c3-45af-aad6-26743703d25e-istio-envoy\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2\" (UID: \"ef78c2a5-79c3-45af-aad6-26743703d25e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2" Oct 30 19:48:36.292953 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:36.292935 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef78c2a5-79c3-45af-aad6-26743703d25e-istio-podinfo\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2\" (UID: \"ef78c2a5-79c3-45af-aad6-26743703d25e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2" Oct 30 19:48:36.353640 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:36.353609 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef78c2a5-79c3-45af-aad6-26743703d25e-istio-token\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2\" (UID: \"ef78c2a5-79c3-45af-aad6-26743703d25e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2" Oct 30 19:48:36.353829 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:36.353806 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxz7f\" (UniqueName: \"kubernetes.io/projected/ef78c2a5-79c3-45af-aad6-26743703d25e-kube-api-access-fxz7f\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2\" (UID: \"ef78c2a5-79c3-45af-aad6-26743703d25e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2" Oct 30 19:48:36.454883 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:36.454794 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2" Oct 30 19:48:36.886226 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:36.886197 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2"] Oct 30 19:48:36.886564 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:48:36.886537 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef78c2a5_79c3_45af_aad6_26743703d25e.slice/crio-8bf2e5f38cf38b5a2592045f93a55a58af2280d6feabd58b234fc267e75fdcf2 WatchSource:0}: Error finding container 8bf2e5f38cf38b5a2592045f93a55a58af2280d6feabd58b234fc267e75fdcf2: Status 404 returned error can't find the container with id 8bf2e5f38cf38b5a2592045f93a55a58af2280d6feabd58b234fc267e75fdcf2 Oct 30 19:48:36.888873 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:36.888841 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:48:36.888977 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:36.888907 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:48:36.888977 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:36.888937 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:48:37.537945 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:37.537911 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2" event={"ID":"ef78c2a5-79c3-45af-aad6-26743703d25e","Type":"ContainerStarted","Data":"4b4a2c10877b8b2e5492486613299b75a74d389864a2b6fbfda14e4139ff64f7"} Oct 30 19:48:37.538467 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:37.538008 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2" event={"ID":"ef78c2a5-79c3-45af-aad6-26743703d25e","Type":"ContainerStarted","Data":"8bf2e5f38cf38b5a2592045f93a55a58af2280d6feabd58b234fc267e75fdcf2"} Oct 30 19:48:37.773397 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:37.773304 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2" podStartSLOduration=1.773283632 podStartE2EDuration="1.773283632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:37.711444279 +0000 UTC m=+1421.229759736" watchObservedRunningTime="2026-10-30 19:48:37.773283632 +0000 UTC m=+1421.291599090" Oct 30 19:48:38.474376 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:38.456705 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2" Oct 30 19:48:38.475843 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:38.475802 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 19:48:38.476008 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:38.475892 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2" podUID="ef78c2a5-79c3-45af-aad6-26743703d25e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 19:48:39.468377 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:39.466799 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 19:48:39.468377 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:39.466880 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2" podUID="ef78c2a5-79c3-45af-aad6-26743703d25e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 19:48:40.463374 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:40.459137 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 19:48:40.463374 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:40.459214 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2" podUID="ef78c2a5-79c3-45af-aad6-26743703d25e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 19:48:41.469377 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:41.468752 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 19:48:41.469377 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:41.468815 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2" podUID="ef78c2a5-79c3-45af-aad6-26743703d25e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 19:48:42.461374 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:42.456749 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 19:48:42.461374 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:42.456826 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2" podUID="ef78c2a5-79c3-45af-aad6-26743703d25e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 19:48:43.221315 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:48:43.221272 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:48:43.455808 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:43.455768 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 19:48:43.456048 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:43.455828 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2" podUID="ef78c2a5-79c3-45af-aad6-26743703d25e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 19:48:45.496522 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:45.496470 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:48:45.497119 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:45.496545 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2" podUID="ef78c2a5-79c3-45af-aad6-26743703d25e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:48:45.691023 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:45.690982 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2" Oct 30 19:48:45.692453 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:45.692419 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2" Oct 30 19:48:45.693239 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:45.693216 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2" Oct 30 19:48:51.284692 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:51.284656 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9"] Oct 30 19:48:51.288645 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:51.288622 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9" Oct 30 19:48:51.371422 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:51.371374 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bbd09dd-3324-4220-ad59-381beadfdf38-istio-podinfo\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9\" (UID: \"5bbd09dd-3324-4220-ad59-381beadfdf38\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9" Oct 30 19:48:51.371713 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:51.371683 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bbd09dd-3324-4220-ad59-381beadfdf38-istiod-ca-cert\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9\" (UID: \"5bbd09dd-3324-4220-ad59-381beadfdf38\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9" Oct 30 19:48:51.371853 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:51.371739 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bbd09dd-3324-4220-ad59-381beadfdf38-istio-data\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9\" (UID: \"5bbd09dd-3324-4220-ad59-381beadfdf38\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9" Oct 30 19:48:51.371938 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:51.371918 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bbd09dd-3324-4220-ad59-381beadfdf38-istio-envoy\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9\" (UID: \"5bbd09dd-3324-4220-ad59-381beadfdf38\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9" Oct 30 19:48:51.372007 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:51.371948 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28mk2\" (UniqueName: \"kubernetes.io/projected/5bbd09dd-3324-4220-ad59-381beadfdf38-kube-api-access-28mk2\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9\" (UID: \"5bbd09dd-3324-4220-ad59-381beadfdf38\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9" Oct 30 19:48:51.372128 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:51.372104 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bbd09dd-3324-4220-ad59-381beadfdf38-istio-token\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9\" (UID: \"5bbd09dd-3324-4220-ad59-381beadfdf38\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9" Oct 30 19:48:51.372198 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:51.372146 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bbd09dd-3324-4220-ad59-381beadfdf38-workload-certs\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9\" (UID: \"5bbd09dd-3324-4220-ad59-381beadfdf38\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9" Oct 30 19:48:51.372246 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:51.372229 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bbd09dd-3324-4220-ad59-381beadfdf38-workload-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9\" (UID: \"5bbd09dd-3324-4220-ad59-381beadfdf38\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9" Oct 30 19:48:51.372299 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:51.372286 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bbd09dd-3324-4220-ad59-381beadfdf38-credential-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9\" (UID: \"5bbd09dd-3324-4220-ad59-381beadfdf38\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9" Oct 30 19:48:51.415935 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:51.415901 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9"] Oct 30 19:48:51.473183 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:51.473146 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bbd09dd-3324-4220-ad59-381beadfdf38-istio-envoy\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9\" (UID: \"5bbd09dd-3324-4220-ad59-381beadfdf38\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9" Oct 30 19:48:51.473183 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:51.473182 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28mk2\" (UniqueName: \"kubernetes.io/projected/5bbd09dd-3324-4220-ad59-381beadfdf38-kube-api-access-28mk2\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9\" (UID: \"5bbd09dd-3324-4220-ad59-381beadfdf38\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9" Oct 30 19:48:51.473485 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:51.473232 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bbd09dd-3324-4220-ad59-381beadfdf38-istio-token\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9\" (UID: \"5bbd09dd-3324-4220-ad59-381beadfdf38\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9" Oct 30 19:48:51.473485 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:51.473250 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bbd09dd-3324-4220-ad59-381beadfdf38-workload-certs\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9\" (UID: \"5bbd09dd-3324-4220-ad59-381beadfdf38\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9" Oct 30 19:48:51.473485 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:51.473275 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bbd09dd-3324-4220-ad59-381beadfdf38-workload-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9\" (UID: \"5bbd09dd-3324-4220-ad59-381beadfdf38\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9" Oct 30 19:48:51.473485 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:51.473299 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bbd09dd-3324-4220-ad59-381beadfdf38-credential-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9\" (UID: \"5bbd09dd-3324-4220-ad59-381beadfdf38\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9" Oct 30 19:48:51.473485 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:51.473370 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bbd09dd-3324-4220-ad59-381beadfdf38-istio-podinfo\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9\" (UID: \"5bbd09dd-3324-4220-ad59-381beadfdf38\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9" Oct 30 19:48:51.473485 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:51.473402 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bbd09dd-3324-4220-ad59-381beadfdf38-istiod-ca-cert\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9\" (UID: \"5bbd09dd-3324-4220-ad59-381beadfdf38\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9" Oct 30 19:48:51.473485 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:51.473442 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bbd09dd-3324-4220-ad59-381beadfdf38-istio-data\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9\" (UID: \"5bbd09dd-3324-4220-ad59-381beadfdf38\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9" Oct 30 19:48:51.473778 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:51.473708 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bbd09dd-3324-4220-ad59-381beadfdf38-workload-certs\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9\" (UID: \"5bbd09dd-3324-4220-ad59-381beadfdf38\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9" Oct 30 19:48:51.473879 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:51.473855 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bbd09dd-3324-4220-ad59-381beadfdf38-istio-data\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9\" (UID: \"5bbd09dd-3324-4220-ad59-381beadfdf38\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9" Oct 30 19:48:51.473992 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:51.473920 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bbd09dd-3324-4220-ad59-381beadfdf38-workload-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9\" (UID: \"5bbd09dd-3324-4220-ad59-381beadfdf38\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9" Oct 30 19:48:51.474063 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:51.474043 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bbd09dd-3324-4220-ad59-381beadfdf38-credential-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9\" (UID: \"5bbd09dd-3324-4220-ad59-381beadfdf38\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9" Oct 30 19:48:51.474315 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:51.474290 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bbd09dd-3324-4220-ad59-381beadfdf38-istiod-ca-cert\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9\" (UID: \"5bbd09dd-3324-4220-ad59-381beadfdf38\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9" Oct 30 19:48:51.475530 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:51.475510 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bbd09dd-3324-4220-ad59-381beadfdf38-istio-envoy\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9\" (UID: \"5bbd09dd-3324-4220-ad59-381beadfdf38\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9" Oct 30 19:48:51.475915 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:51.475896 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bbd09dd-3324-4220-ad59-381beadfdf38-istio-podinfo\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9\" (UID: \"5bbd09dd-3324-4220-ad59-381beadfdf38\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9" Oct 30 19:48:51.535389 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:51.535289 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bbd09dd-3324-4220-ad59-381beadfdf38-istio-token\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9\" (UID: \"5bbd09dd-3324-4220-ad59-381beadfdf38\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9" Oct 30 19:48:51.535555 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:51.535425 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28mk2\" (UniqueName: \"kubernetes.io/projected/5bbd09dd-3324-4220-ad59-381beadfdf38-kube-api-access-28mk2\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9\" (UID: \"5bbd09dd-3324-4220-ad59-381beadfdf38\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9" Oct 30 19:48:51.602998 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:51.602968 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9" Oct 30 19:48:52.132398 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:52.132369 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9"] Oct 30 19:48:52.133197 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:48:52.133172 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bbd09dd_3324_4220_ad59_381beadfdf38.slice/crio-c23bfb4394e148ac97d705ce7546bb6d9099ffeddfb77f93e61b7161c2700cab WatchSource:0}: Error finding container c23bfb4394e148ac97d705ce7546bb6d9099ffeddfb77f93e61b7161c2700cab: Status 404 returned error can't find the container with id c23bfb4394e148ac97d705ce7546bb6d9099ffeddfb77f93e61b7161c2700cab Oct 30 19:48:52.135605 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:52.135568 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:48:52.135683 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:52.135637 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:48:52.135683 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:52.135668 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:48:52.687193 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:52.687153 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9" event={"ID":"5bbd09dd-3324-4220-ad59-381beadfdf38","Type":"ContainerStarted","Data":"a3cdc21e06f9a2f9218e2ffdd61e7e2b50e93bc9abff5c479de3a4f32b8e0f67"} Oct 30 19:48:52.687617 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:52.687210 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9" event={"ID":"5bbd09dd-3324-4220-ad59-381beadfdf38","Type":"ContainerStarted","Data":"c23bfb4394e148ac97d705ce7546bb6d9099ffeddfb77f93e61b7161c2700cab"} Oct 30 19:48:52.810918 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:52.810851 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9" podStartSLOduration=1.810832467 podStartE2EDuration="1.810832467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:52.810102443 +0000 UTC m=+1436.328417900" watchObservedRunningTime="2026-10-30 19:48:52.810832467 +0000 UTC m=+1436.329147924" Oct 30 19:48:53.603933 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:53.603893 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9" Oct 30 19:48:53.605721 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:53.605691 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 19:48:53.605880 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:53.605766 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9" podUID="5bbd09dd-3324-4220-ad59-381beadfdf38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 19:48:54.603838 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:54.603802 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 19:48:54.604205 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:54.603853 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9" podUID="5bbd09dd-3324-4220-ad59-381beadfdf38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 19:48:55.603737 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:55.603701 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 19:48:55.603920 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:55.603755 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9" podUID="5bbd09dd-3324-4220-ad59-381beadfdf38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 19:48:56.603555 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:56.603524 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 19:48:56.603732 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:56.603583 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9" podUID="5bbd09dd-3324-4220-ad59-381beadfdf38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 19:48:57.611377 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:57.606988 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 19:48:57.611377 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:57.607070 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9" podUID="5bbd09dd-3324-4220-ad59-381beadfdf38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 19:48:58.349545 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:58.349490 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bbfa07/distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2" podUID="f562f81a-ebd3-4efa-889d-bf4a757527fe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:48:58.521379 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:58.520415 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a74d15/premium-w0-a74d15-kserve-864f45fdfb-949gm" podUID="eb0ced80-4c4f-41dc-883c-b1a21e0f11f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 30 19:48:58.604435 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:58.603923 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 19:48:58.604435 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:58.604020 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9" podUID="5bbd09dd-3324-4220-ad59-381beadfdf38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 19:48:58.657372 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:58.657217 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-98a19b/simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r" podUID="1f35e32e-1f09-413d-a364-14b8ac1019f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:48:58.707374 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:58.707221 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ec609/premium-w4-4ec609-kserve-69f7c88bf7-rzmmt" podUID="fb6be431-0a6c-4d81-b365-d7537f5d385a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:48:59.604428 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:59.604387 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 19:48:59.604596 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:48:59.604453 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9" podUID="5bbd09dd-3324-4220-ad59-381beadfdf38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 19:49:01.608835 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:01.608797 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:49:01.609361 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:01.608853 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9" podUID="5bbd09dd-3324-4220-ad59-381beadfdf38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:49:01.615978 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:01.615949 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9" Oct 30 19:49:01.616836 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:01.616816 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9" Oct 30 19:49:01.617477 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:01.617457 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9" Oct 30 19:49:13.220039 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:49:13.219995 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:49:28.350571 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:28.350505 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bbfa07/distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2" podUID="f562f81a-ebd3-4efa-889d-bf4a757527fe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:49:28.405505 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:28.405451 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a74d15/premium-w0-a74d15-kserve-864f45fdfb-949gm" podUID="eb0ced80-4c4f-41dc-883c-b1a21e0f11f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:49:28.642883 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:28.642782 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bbfa07/distinct-w3-bbfa07-kserve-67bb47494b-t8t7l" podUID="16def41a-a612-4a4c-ada9-bb170b066af2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:49:28.659861 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:28.655432 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-98a19b/simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r" podUID="1f35e32e-1f09-413d-a364-14b8ac1019f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:49:28.705208 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:28.705165 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ec609/premium-w4-4ec609-kserve-69f7c88bf7-rzmmt" podUID="fb6be431-0a6c-4d81-b365-d7537f5d385a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:49:28.750265 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:28.749444 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bbfa07/premium-w3-bbfa07-kserve-5675444c9-x42d7" podUID="e0b75746-c06c-407d-8f61-c795c519aad6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:49:28.766274 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:28.766216 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a74d15/distinct-2-w0-a74d15-kserve-846bdb9948-ht54r" podUID="f70232fa-7106-4ec6-8c8e-277c4ce4c719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:49:28.878643 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:28.878459 4292 patch_prober.go:29] interesting pod/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:49:28.878643 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:28.878567 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml" podUID="69d11c12-c4cb-452e-90cc-4cbe03333951" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:49:29.433815 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:29.433769 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a74d15/premium-w0-a74d15-kserve-864f45fdfb-949gm" podUID="eb0ced80-4c4f-41dc-883c-b1a21e0f11f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 30 19:49:29.435132 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:29.435104 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a74d15/premium-w0-a74d15-kserve-864f45fdfb-949gm" Oct 30 19:49:29.458001 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:29.443961 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bbfa07/distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2" podUID="f562f81a-ebd3-4efa-889d-bf4a757527fe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:49:29.458001 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:29.444286 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bbfa07/distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2" Oct 30 19:49:29.561607 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:29.545827 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a74d15/premium-w0-a74d15-kserve-864f45fdfb-949gm" Oct 30 19:49:29.565384 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:29.562988 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bbfa07/distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2" Oct 30 19:49:29.648400 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:29.647230 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bbfa07/distinct-w3-bbfa07-kserve-67bb47494b-t8t7l" podUID="16def41a-a612-4a4c-ada9-bb170b066af2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 30 19:49:29.648400 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:29.648020 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bbfa07/distinct-w3-bbfa07-kserve-67bb47494b-t8t7l" Oct 30 19:49:29.817966 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:29.776682 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bbfa07/premium-w3-bbfa07-kserve-5675444c9-x42d7" podUID="e0b75746-c06c-407d-8f61-c795c519aad6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 19:49:29.817966 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:29.777485 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bbfa07/premium-w3-bbfa07-kserve-5675444c9-x42d7" Oct 30 19:49:29.817966 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:29.779824 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a74d15/distinct-2-w0-a74d15-kserve-846bdb9948-ht54r" podUID="f70232fa-7106-4ec6-8c8e-277c4ce4c719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:49:29.817966 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:29.780169 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a74d15/distinct-2-w0-a74d15-kserve-846bdb9948-ht54r" Oct 30 19:49:30.714382 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:30.703107 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-98a19b/simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r" podUID="1f35e32e-1f09-413d-a364-14b8ac1019f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:49:30.714382 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:30.705681 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a74d15/premium-w0-a74d15-kserve-864f45fdfb-949gm" podUID="eb0ced80-4c4f-41dc-883c-b1a21e0f11f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:49:30.714382 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:30.706112 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bbfa07/distinct-w3-bbfa07-kserve-67bb47494b-t8t7l" podUID="16def41a-a612-4a4c-ada9-bb170b066af2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:49:30.790365 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:30.790222 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bbfa07/premium-w3-bbfa07-kserve-5675444c9-x42d7" podUID="e0b75746-c06c-407d-8f61-c795c519aad6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:49:30.956298 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:30.956256 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a74d15/distinct-2-w0-a74d15-kserve-846bdb9948-ht54r" Oct 30 19:49:30.991424 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:30.991252 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bbfa07/distinct-w3-bbfa07-kserve-67bb47494b-t8t7l" Oct 30 19:49:31.021422 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:31.008942 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bbfa07/premium-w3-bbfa07-kserve-5675444c9-x42d7" Oct 30 19:49:32.462413 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:32.462374 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t"] Oct 30 19:49:32.462879 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:32.462646 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t" podUID="5de2e40c-7f13-4b3c-ab43-5f3a5c351d50" containerName="istio-proxy" containerID="cri-o://cd12ef940af7322aff818a272955ff70b4b9fd4d11fef6a7e156416f34129173" gracePeriod=30 Oct 30 19:49:37.378154 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:37.378119 4292 patch_prober.go:29] interesting pod/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:49:37.378573 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:37.378199 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t" podUID="5de2e40c-7f13-4b3c-ab43-5f3a5c351d50" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:49:37.726868 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:37.726846 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t" Oct 30 19:49:37.863955 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:37.863920 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50-credential-socket\") pod \"5de2e40c-7f13-4b3c-ab43-5f3a5c351d50\" (UID: \"5de2e40c-7f13-4b3c-ab43-5f3a5c351d50\") " Oct 30 19:49:37.864124 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:37.863962 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50-istio-token\" (UniqueName: \"kubernetes.io/projected/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50-istio-token\") pod \"5de2e40c-7f13-4b3c-ab43-5f3a5c351d50\" (UID: \"5de2e40c-7f13-4b3c-ab43-5f3a5c351d50\") " Oct 30 19:49:37.864124 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:37.864042 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50-istio-podinfo\") pod \"5de2e40c-7f13-4b3c-ab43-5f3a5c351d50\" (UID: \"5de2e40c-7f13-4b3c-ab43-5f3a5c351d50\") " Oct 30 19:49:37.864124 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:37.864069 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50-istio-data\") pod \"5de2e40c-7f13-4b3c-ab43-5f3a5c351d50\" (UID: \"5de2e40c-7f13-4b3c-ab43-5f3a5c351d50\") " Oct 30 19:49:37.864124 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:37.864111 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50-workload-certs\") pod \"5de2e40c-7f13-4b3c-ab43-5f3a5c351d50\" (UID: \"5de2e40c-7f13-4b3c-ab43-5f3a5c351d50\") " Oct 30 19:49:37.864344 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:37.864160 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50-credential-socket" pod "5de2e40c-7f13-4b3c-ab43-5f3a5c351d50" (UID: "5de2e40c-7f13-4b3c-ab43-5f3a5c351d50"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:49:37.864344 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:37.864192 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50-istio-envoy\") pod \"5de2e40c-7f13-4b3c-ab43-5f3a5c351d50\" (UID: \"5de2e40c-7f13-4b3c-ab43-5f3a5c351d50\") " Oct 30 19:49:37.864344 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:37.864222 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50-workload-socket\") pod \"5de2e40c-7f13-4b3c-ab43-5f3a5c351d50\" (UID: \"5de2e40c-7f13-4b3c-ab43-5f3a5c351d50\") " Oct 30 19:49:37.864344 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:37.864280 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50-istiod-ca-cert\") pod \"5de2e40c-7f13-4b3c-ab43-5f3a5c351d50\" (UID: \"5de2e40c-7f13-4b3c-ab43-5f3a5c351d50\") " Oct 30 19:49:37.864344 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:37.864346 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50-kube-api-access-b4ss9\" (UniqueName: \"kubernetes.io/projected/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50-kube-api-access-b4ss9\") pod \"5de2e40c-7f13-4b3c-ab43-5f3a5c351d50\" (UID: \"5de2e40c-7f13-4b3c-ab43-5f3a5c351d50\") " Oct 30 19:49:37.864645 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:37.864399 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50-istio-data" pod "5de2e40c-7f13-4b3c-ab43-5f3a5c351d50" (UID: "5de2e40c-7f13-4b3c-ab43-5f3a5c351d50"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:49:37.864645 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:37.864455 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50-workload-certs" pod "5de2e40c-7f13-4b3c-ab43-5f3a5c351d50" (UID: "5de2e40c-7f13-4b3c-ab43-5f3a5c351d50"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:49:37.864645 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:37.864551 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50-workload-socket" pod "5de2e40c-7f13-4b3c-ab43-5f3a5c351d50" (UID: "5de2e40c-7f13-4b3c-ab43-5f3a5c351d50"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:49:37.864849 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:37.864810 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50-istiod-ca-cert" pod "5de2e40c-7f13-4b3c-ab43-5f3a5c351d50" (UID: "5de2e40c-7f13-4b3c-ab43-5f3a5c351d50"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:49:37.864993 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:37.864948 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50-workload-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:49:37.864993 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:37.864970 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50-credential-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:49:37.864993 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:37.864984 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50-istio-data\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:49:37.865170 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:37.865000 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50-workload-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:49:37.866614 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:37.866584 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50-istio-envoy" pod "5de2e40c-7f13-4b3c-ab43-5f3a5c351d50" (UID: "5de2e40c-7f13-4b3c-ab43-5f3a5c351d50"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:49:37.866614 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:37.866606 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50-kube-api-access-b4ss9" pod "5de2e40c-7f13-4b3c-ab43-5f3a5c351d50" (UID: "5de2e40c-7f13-4b3c-ab43-5f3a5c351d50"). InnerVolumeSpecName "kube-api-access-b4ss9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:49:37.866755 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:37.866646 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50-istio-token" pod "5de2e40c-7f13-4b3c-ab43-5f3a5c351d50" (UID: "5de2e40c-7f13-4b3c-ab43-5f3a5c351d50"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:49:37.866755 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:37.866672 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50-istio-podinfo" pod "5de2e40c-7f13-4b3c-ab43-5f3a5c351d50" (UID: "5de2e40c-7f13-4b3c-ab43-5f3a5c351d50"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:49:37.966467 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:37.966393 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50-istiod-ca-cert\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:49:37.966467 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:37.966419 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4ss9\" (UniqueName: \"kubernetes.io/projected/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50-kube-api-access-b4ss9\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:49:37.966467 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:37.966429 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50-istio-token\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:49:37.966467 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:37.966444 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50-istio-podinfo\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:49:37.966467 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:37.966452 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50-istio-envoy\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:49:38.178088 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:38.178053 4292 generic.go:356] "Generic (PLEG): container finished" podID="5de2e40c-7f13-4b3c-ab43-5f3a5c351d50" containerID="cd12ef940af7322aff818a272955ff70b4b9fd4d11fef6a7e156416f34129173" exitCode=0 Oct 30 19:49:38.178265 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:38.178118 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t" Oct 30 19:49:38.178265 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:38.178134 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t" event={"ID":"5de2e40c-7f13-4b3c-ab43-5f3a5c351d50","Type":"ContainerDied","Data":"cd12ef940af7322aff818a272955ff70b4b9fd4d11fef6a7e156416f34129173"} Oct 30 19:49:38.178265 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:38.178195 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t" event={"ID":"5de2e40c-7f13-4b3c-ab43-5f3a5c351d50","Type":"ContainerDied","Data":"cafa8802e6ad0edeb44b76d76255b44d4c6111371929f5f6c085e79b4335cd02"} Oct 30 19:49:38.178265 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:38.178218 4292 scope.go:122] "RemoveContainer" containerID="cd12ef940af7322aff818a272955ff70b4b9fd4d11fef6a7e156416f34129173" Oct 30 19:49:38.187967 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:38.187947 4292 scope.go:122] "RemoveContainer" containerID="cd12ef940af7322aff818a272955ff70b4b9fd4d11fef6a7e156416f34129173" Oct 30 19:49:38.188251 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:49:38.188233 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd12ef940af7322aff818a272955ff70b4b9fd4d11fef6a7e156416f34129173\": container with ID starting with cd12ef940af7322aff818a272955ff70b4b9fd4d11fef6a7e156416f34129173 not found: ID does not exist" containerID="cd12ef940af7322aff818a272955ff70b4b9fd4d11fef6a7e156416f34129173" Oct 30 19:49:38.188327 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:38.188260 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd12ef940af7322aff818a272955ff70b4b9fd4d11fef6a7e156416f34129173"} err="failed to get container status \"cd12ef940af7322aff818a272955ff70b4b9fd4d11fef6a7e156416f34129173\": rpc error: code = NotFound desc = could not find container \"cd12ef940af7322aff818a272955ff70b4b9fd4d11fef6a7e156416f34129173\": container with ID starting with cd12ef940af7322aff818a272955ff70b4b9fd4d11fef6a7e156416f34129173 not found: ID does not exist" Oct 30 19:49:38.351389 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:38.351344 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t"] Oct 30 19:49:38.402989 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:38.402958 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-52d5t"] Oct 30 19:49:38.588964 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:38.588931 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5de2e40c-7f13-4b3c-ab43-5f3a5c351d50" path="/var/lib/kubelet/pods/5de2e40c-7f13-4b3c-ab43-5f3a5c351d50/volumes" Oct 30 19:49:43.221982 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:49:43.221897 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:49:47.624969 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:47.624932 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4"] Oct 30 19:49:47.627524 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:47.625475 4292 cpu_manager.go:414] "containerMap: removing container" podUID="5de2e40c-7f13-4b3c-ab43-5f3a5c351d50" containerName="istio-proxy" Oct 30 19:49:47.627524 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:47.625488 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5de2e40c-7f13-4b3c-ab43-5f3a5c351d50" containerName="istio-proxy" Oct 30 19:49:47.627524 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:47.625560 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="5de2e40c-7f13-4b3c-ab43-5f3a5c351d50" containerName="istio-proxy" Oct 30 19:49:47.628655 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:47.628637 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4" Oct 30 19:49:47.682335 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:47.682299 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-4ec609-openshift-default-dockercfg-4txpk\"" Oct 30 19:49:47.736063 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:47.736016 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4"] Oct 30 19:49:47.792044 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:47.792002 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d8780e4-df03-426f-8981-f355e0430b39-credential-socket\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4\" (UID: \"6d8780e4-df03-426f-8981-f355e0430b39\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4" Oct 30 19:49:47.792243 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:47.792064 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d8780e4-df03-426f-8981-f355e0430b39-istiod-ca-cert\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4\" (UID: \"6d8780e4-df03-426f-8981-f355e0430b39\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4" Oct 30 19:49:47.792243 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:47.792150 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d8780e4-df03-426f-8981-f355e0430b39-istio-podinfo\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4\" (UID: \"6d8780e4-df03-426f-8981-f355e0430b39\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4" Oct 30 19:49:47.792243 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:47.792190 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d8780e4-df03-426f-8981-f355e0430b39-istio-token\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4\" (UID: \"6d8780e4-df03-426f-8981-f355e0430b39\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4" Oct 30 19:49:47.792243 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:47.792221 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d8780e4-df03-426f-8981-f355e0430b39-workload-certs\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4\" (UID: \"6d8780e4-df03-426f-8981-f355e0430b39\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4" Oct 30 19:49:47.792484 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:47.792255 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d8780e4-df03-426f-8981-f355e0430b39-istio-envoy\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4\" (UID: \"6d8780e4-df03-426f-8981-f355e0430b39\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4" Oct 30 19:49:47.792484 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:47.792346 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d8780e4-df03-426f-8981-f355e0430b39-workload-socket\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4\" (UID: \"6d8780e4-df03-426f-8981-f355e0430b39\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4" Oct 30 19:49:47.792484 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:47.792409 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9kr7\" (UniqueName: \"kubernetes.io/projected/6d8780e4-df03-426f-8981-f355e0430b39-kube-api-access-p9kr7\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4\" (UID: \"6d8780e4-df03-426f-8981-f355e0430b39\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4" Oct 30 19:49:47.792591 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:47.792512 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d8780e4-df03-426f-8981-f355e0430b39-istio-data\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4\" (UID: \"6d8780e4-df03-426f-8981-f355e0430b39\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4" Oct 30 19:49:47.894298 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:47.894207 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d8780e4-df03-426f-8981-f355e0430b39-istio-data\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4\" (UID: \"6d8780e4-df03-426f-8981-f355e0430b39\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4" Oct 30 19:49:47.894492 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:47.894333 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d8780e4-df03-426f-8981-f355e0430b39-credential-socket\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4\" (UID: \"6d8780e4-df03-426f-8981-f355e0430b39\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4" Oct 30 19:49:47.894492 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:47.894370 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d8780e4-df03-426f-8981-f355e0430b39-istiod-ca-cert\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4\" (UID: \"6d8780e4-df03-426f-8981-f355e0430b39\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4" Oct 30 19:49:47.894492 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:47.894403 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d8780e4-df03-426f-8981-f355e0430b39-istio-podinfo\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4\" (UID: \"6d8780e4-df03-426f-8981-f355e0430b39\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4" Oct 30 19:49:47.894492 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:47.894425 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d8780e4-df03-426f-8981-f355e0430b39-istio-token\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4\" (UID: \"6d8780e4-df03-426f-8981-f355e0430b39\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4" Oct 30 19:49:47.894492 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:47.894453 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d8780e4-df03-426f-8981-f355e0430b39-workload-certs\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4\" (UID: \"6d8780e4-df03-426f-8981-f355e0430b39\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4" Oct 30 19:49:47.894492 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:47.894482 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d8780e4-df03-426f-8981-f355e0430b39-istio-envoy\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4\" (UID: \"6d8780e4-df03-426f-8981-f355e0430b39\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4" Oct 30 19:49:47.894858 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:47.894513 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d8780e4-df03-426f-8981-f355e0430b39-workload-socket\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4\" (UID: \"6d8780e4-df03-426f-8981-f355e0430b39\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4" Oct 30 19:49:47.894858 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:47.894681 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d8780e4-df03-426f-8981-f355e0430b39-istio-data\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4\" (UID: \"6d8780e4-df03-426f-8981-f355e0430b39\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4" Oct 30 19:49:47.894858 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:47.894727 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d8780e4-df03-426f-8981-f355e0430b39-credential-socket\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4\" (UID: \"6d8780e4-df03-426f-8981-f355e0430b39\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4" Oct 30 19:49:47.894858 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:47.894735 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9kr7\" (UniqueName: \"kubernetes.io/projected/6d8780e4-df03-426f-8981-f355e0430b39-kube-api-access-p9kr7\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4\" (UID: \"6d8780e4-df03-426f-8981-f355e0430b39\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4" Oct 30 19:49:47.894858 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:47.894797 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d8780e4-df03-426f-8981-f355e0430b39-workload-socket\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4\" (UID: \"6d8780e4-df03-426f-8981-f355e0430b39\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4" Oct 30 19:49:47.895151 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:47.895130 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d8780e4-df03-426f-8981-f355e0430b39-workload-certs\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4\" (UID: \"6d8780e4-df03-426f-8981-f355e0430b39\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4" Oct 30 19:49:47.895254 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:47.895222 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d8780e4-df03-426f-8981-f355e0430b39-istiod-ca-cert\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4\" (UID: \"6d8780e4-df03-426f-8981-f355e0430b39\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4" Oct 30 19:49:47.896848 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:47.896827 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d8780e4-df03-426f-8981-f355e0430b39-istio-envoy\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4\" (UID: \"6d8780e4-df03-426f-8981-f355e0430b39\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4" Oct 30 19:49:47.897226 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:47.897205 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d8780e4-df03-426f-8981-f355e0430b39-istio-podinfo\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4\" (UID: \"6d8780e4-df03-426f-8981-f355e0430b39\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4" Oct 30 19:49:47.959834 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:47.959801 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d8780e4-df03-426f-8981-f355e0430b39-istio-token\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4\" (UID: \"6d8780e4-df03-426f-8981-f355e0430b39\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4" Oct 30 19:49:47.959982 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:47.959886 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9kr7\" (UniqueName: \"kubernetes.io/projected/6d8780e4-df03-426f-8981-f355e0430b39-kube-api-access-p9kr7\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4\" (UID: \"6d8780e4-df03-426f-8981-f355e0430b39\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4" Oct 30 19:49:48.242469 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:48.242371 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4" Oct 30 19:49:48.704739 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:49:48.704536 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d8780e4_df03_426f_8981_f355e0430b39.slice/crio-1c3748213065706173b84e157163efa2c9e26241f4285bded7537a3bfca0baf0 WatchSource:0}: Error finding container 1c3748213065706173b84e157163efa2c9e26241f4285bded7537a3bfca0baf0: Status 404 returned error can't find the container with id 1c3748213065706173b84e157163efa2c9e26241f4285bded7537a3bfca0baf0 Oct 30 19:49:48.706367 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:48.706267 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4"] Oct 30 19:49:48.708153 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:48.708116 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:49:48.708343 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:48.708319 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:49:48.708527 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:48.708506 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:49:49.246406 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:49.245160 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4" event={"ID":"6d8780e4-df03-426f-8981-f355e0430b39","Type":"ContainerStarted","Data":"249c54f8a9b0e31bc04c97a3b9318efdbb84fb88a517903daaff589c964d4167"} Oct 30 19:49:49.246406 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:49.245222 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4" event={"ID":"6d8780e4-df03-426f-8981-f355e0430b39","Type":"ContainerStarted","Data":"1c3748213065706173b84e157163efa2c9e26241f4285bded7537a3bfca0baf0"} Oct 30 19:49:49.371524 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:49.371455 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4" podStartSLOduration=2.371434186 podStartE2EDuration="2.371434186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:49.368114237 +0000 UTC m=+1492.886429706" watchObservedRunningTime="2026-10-30 19:49:49.371434186 +0000 UTC m=+1492.889749645" Oct 30 19:49:50.246021 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:50.245290 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4" Oct 30 19:49:50.262377 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:50.261261 4292 patch_prober.go:29] interesting pod/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 19:49:50.262377 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:50.261365 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4" podUID="6d8780e4-df03-426f-8981-f355e0430b39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 19:49:51.242984 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:51.242945 4292 patch_prober.go:29] interesting pod/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 19:49:51.243180 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:51.243006 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4" podUID="6d8780e4-df03-426f-8981-f355e0430b39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 19:49:52.243770 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:52.243733 4292 patch_prober.go:29] interesting pod/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 19:49:52.244252 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:52.243796 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4" podUID="6d8780e4-df03-426f-8981-f355e0430b39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 19:49:53.243110 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:53.243072 4292 patch_prober.go:29] interesting pod/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 19:49:53.243290 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:53.243131 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4" podUID="6d8780e4-df03-426f-8981-f355e0430b39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 19:49:54.242863 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:54.242826 4292 patch_prober.go:29] interesting pod/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 19:49:54.243286 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:54.242912 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4" podUID="6d8780e4-df03-426f-8981-f355e0430b39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 19:49:55.242932 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:55.242892 4292 patch_prober.go:29] interesting pod/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 19:49:55.243485 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:55.242950 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4" podUID="6d8780e4-df03-426f-8981-f355e0430b39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 19:49:56.248770 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:56.248737 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4" Oct 30 19:49:56.249263 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:56.249238 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4" Oct 30 19:49:56.250122 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:49:56.250106 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4" Oct 30 19:50:13.224616 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:50:13.224570 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:50:38.448043 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:50:38.448001 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cf283d-b99794875-llrmk"] Oct 30 19:50:38.454834 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:50:38.454413 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cf283d-b99794875-llrmk" Oct 30 19:50:38.502335 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:50:38.502300 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cf283d-b99794875-llrmk"] Oct 30 19:50:38.558811 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:50:38.558775 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-cf283d-dockercfg-4wtl6\"" Oct 30 19:50:38.559025 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:50:38.558869 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-cf283d\"" Oct 30 19:50:38.609407 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:50:38.607240 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk9nr\" (UniqueName: \"kubernetes.io/projected/730c37c8-3814-42cc-b40e-c557279ad8ca-kube-api-access-wk9nr\") pod \"payload-pre-processing-e2e-api-a-cf283d-b99794875-llrmk\" (UID: \"730c37c8-3814-42cc-b40e-c557279ad8ca\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cf283d-b99794875-llrmk" Oct 30 19:50:38.609407 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:50:38.607405 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/730c37c8-3814-42cc-b40e-c557279ad8ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-cf283d-b99794875-llrmk\" (UID: \"730c37c8-3814-42cc-b40e-c557279ad8ca\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cf283d-b99794875-llrmk" Oct 30 19:50:38.711670 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:50:38.711578 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/730c37c8-3814-42cc-b40e-c557279ad8ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-cf283d-b99794875-llrmk\" (UID: \"730c37c8-3814-42cc-b40e-c557279ad8ca\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cf283d-b99794875-llrmk" Oct 30 19:50:38.712953 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:50:38.712929 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk9nr\" (UniqueName: \"kubernetes.io/projected/730c37c8-3814-42cc-b40e-c557279ad8ca-kube-api-access-wk9nr\") pod \"payload-pre-processing-e2e-api-a-cf283d-b99794875-llrmk\" (UID: \"730c37c8-3814-42cc-b40e-c557279ad8ca\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cf283d-b99794875-llrmk" Oct 30 19:50:38.713538 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:50:38.712666 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/730c37c8-3814-42cc-b40e-c557279ad8ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-cf283d-b99794875-llrmk\" (UID: \"730c37c8-3814-42cc-b40e-c557279ad8ca\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cf283d-b99794875-llrmk" Oct 30 19:50:38.813306 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:50:38.813271 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk9nr\" (UniqueName: \"kubernetes.io/projected/730c37c8-3814-42cc-b40e-c557279ad8ca-kube-api-access-wk9nr\") pod \"payload-pre-processing-e2e-api-a-cf283d-b99794875-llrmk\" (UID: \"730c37c8-3814-42cc-b40e-c557279ad8ca\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cf283d-b99794875-llrmk" Oct 30 19:50:39.069740 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:50:39.069708 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cf283d-b99794875-llrmk" Oct 30 19:50:39.445891 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:50:39.445858 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cf283d-b99794875-llrmk"] Oct 30 19:50:39.446568 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:50:39.446535 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod730c37c8_3814_42cc_b40e_c557279ad8ca.slice/crio-1f16ff51eef26bfbee85490dc7de47ccd7cef5dfc00f9e36524586e4d8f828f3 WatchSource:0}: Error finding container 1f16ff51eef26bfbee85490dc7de47ccd7cef5dfc00f9e36524586e4d8f828f3: Status 404 returned error can't find the container with id 1f16ff51eef26bfbee85490dc7de47ccd7cef5dfc00f9e36524586e4d8f828f3 Oct 30 19:50:39.744849 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:50:39.744757 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cf283d-b99794875-llrmk" event={"ID":"730c37c8-3814-42cc-b40e-c557279ad8ca","Type":"ContainerStarted","Data":"3fc5099ae864ecec770d4091c3ead21100c3d89061837b90a3370941e8ce1e7a"} Oct 30 19:50:39.744849 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:50:39.744808 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cf283d-b99794875-llrmk" event={"ID":"730c37c8-3814-42cc-b40e-c557279ad8ca","Type":"ContainerStarted","Data":"1f16ff51eef26bfbee85490dc7de47ccd7cef5dfc00f9e36524586e4d8f828f3"} Oct 30 19:50:39.745288 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:50:39.744970 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cf283d-b99794875-llrmk" Oct 30 19:50:39.865239 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:50:39.865177 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cf283d-b99794875-llrmk" podStartSLOduration=1.865162591 podStartE2EDuration="1.865162591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:39.863059941 +0000 UTC m=+1543.381375398" watchObservedRunningTime="2026-10-30 19:50:39.865162591 +0000 UTC m=+1543.383478048" Oct 30 19:50:43.225714 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:50:43.225673 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:50:50.753516 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:50:50.753487 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cf283d-b99794875-llrmk" Oct 30 19:51:04.253339 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:04.253285 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-111f9c64-759998544b-n4dkv"] Oct 30 19:51:04.264883 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:04.264850 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-111f9c64-759998544b-n4dkv" Oct 30 19:51:04.350378 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:04.337465 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-111f9c64-759998544b-n4dkv"] Oct 30 19:51:04.368383 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:04.365735 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-111f9c64-dockercfg-xkc7k\"" Oct 30 19:51:04.368383 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:04.365867 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-111f9c64\"" Oct 30 19:51:04.456879 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:04.453068 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6ndx\" (UniqueName: \"kubernetes.io/projected/d7eb2b8c-8309-4951-82f7-e0f385da96a5-kube-api-access-q6ndx\") pod \"payload-processing-e2e-mt-111f9c64-759998544b-n4dkv\" (UID: \"d7eb2b8c-8309-4951-82f7-e0f385da96a5\") " pod="openshift-ingress/payload-processing-e2e-mt-111f9c64-759998544b-n4dkv" Oct 30 19:51:04.456879 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:04.453148 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7eb2b8c-8309-4951-82f7-e0f385da96a5-plugins-config-volume\") pod \"payload-processing-e2e-mt-111f9c64-759998544b-n4dkv\" (UID: \"d7eb2b8c-8309-4951-82f7-e0f385da96a5\") " pod="openshift-ingress/payload-processing-e2e-mt-111f9c64-759998544b-n4dkv" Oct 30 19:51:04.554422 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:04.554382 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6ndx\" (UniqueName: \"kubernetes.io/projected/d7eb2b8c-8309-4951-82f7-e0f385da96a5-kube-api-access-q6ndx\") pod \"payload-processing-e2e-mt-111f9c64-759998544b-n4dkv\" (UID: \"d7eb2b8c-8309-4951-82f7-e0f385da96a5\") " pod="openshift-ingress/payload-processing-e2e-mt-111f9c64-759998544b-n4dkv" Oct 30 19:51:04.554647 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:04.554438 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7eb2b8c-8309-4951-82f7-e0f385da96a5-plugins-config-volume\") pod \"payload-processing-e2e-mt-111f9c64-759998544b-n4dkv\" (UID: \"d7eb2b8c-8309-4951-82f7-e0f385da96a5\") " pod="openshift-ingress/payload-processing-e2e-mt-111f9c64-759998544b-n4dkv" Oct 30 19:51:04.555443 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:04.555416 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7eb2b8c-8309-4951-82f7-e0f385da96a5-plugins-config-volume\") pod \"payload-processing-e2e-mt-111f9c64-759998544b-n4dkv\" (UID: \"d7eb2b8c-8309-4951-82f7-e0f385da96a5\") " pod="openshift-ingress/payload-processing-e2e-mt-111f9c64-759998544b-n4dkv" Oct 30 19:51:04.617741 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:04.617709 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6ndx\" (UniqueName: \"kubernetes.io/projected/d7eb2b8c-8309-4951-82f7-e0f385da96a5-kube-api-access-q6ndx\") pod \"payload-processing-e2e-mt-111f9c64-759998544b-n4dkv\" (UID: \"d7eb2b8c-8309-4951-82f7-e0f385da96a5\") " pod="openshift-ingress/payload-processing-e2e-mt-111f9c64-759998544b-n4dkv" Oct 30 19:51:04.904600 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:04.904513 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-111f9c64-759998544b-n4dkv" Oct 30 19:51:05.278776 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:05.278742 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-111f9c64-759998544b-n4dkv"] Oct 30 19:51:05.280120 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:51:05.280092 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7eb2b8c_8309_4951_82f7_e0f385da96a5.slice/crio-629b127329b1a866d687bec8c94513676765c748c7abe5c3dc239cf3955026f0 WatchSource:0}: Error finding container 629b127329b1a866d687bec8c94513676765c748c7abe5c3dc239cf3955026f0: Status 404 returned error can't find the container with id 629b127329b1a866d687bec8c94513676765c748c7abe5c3dc239cf3955026f0 Oct 30 19:51:05.958730 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:05.958695 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-111f9c64-759998544b-n4dkv" event={"ID":"d7eb2b8c-8309-4951-82f7-e0f385da96a5","Type":"ContainerStarted","Data":"490b6f311f94a9548131945131f6568de5ef5ce2b96be1b0aec00cb961cfe71c"} Oct 30 19:51:05.958945 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:05.958743 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-111f9c64-759998544b-n4dkv" event={"ID":"d7eb2b8c-8309-4951-82f7-e0f385da96a5","Type":"ContainerStarted","Data":"629b127329b1a866d687bec8c94513676765c748c7abe5c3dc239cf3955026f0"} Oct 30 19:51:05.958945 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:05.958866 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-111f9c64-759998544b-n4dkv" Oct 30 19:51:06.077928 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:06.077879 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-111f9c64-759998544b-n4dkv" podStartSLOduration=2.077863434 podStartE2EDuration="2.077863434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:06.077327057 +0000 UTC m=+1569.595642493" watchObservedRunningTime="2026-10-30 19:51:06.077863434 +0000 UTC m=+1569.596178893" Oct 30 19:51:13.242590 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:51:13.242501 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:51:14.664386 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:14.664190 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-lvnj7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:51:14.664386 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:14.664310 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lvnj7" podUID="317c18b6-5b6b-4af5-9ba3-3d5ad719d53b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:14.665080 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:14.665052 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-kr46d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:51:14.665287 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:14.665263 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kr46d" podUID="de3476ef-83a0-4ca2-9b76-227ee4b35d7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:14.906889 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:14.906847 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-8nrtp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:51:14.907240 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:14.907205 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8nrtp" podUID="b5978bc5-3ff8-4f4a-b85f-58b737179fee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:17.044380 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:17.024639 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-111f9c64-759998544b-n4dkv" Oct 30 19:51:20.584194 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:20.584154 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4477e2-995c7ff66-8w9p8"] Oct 30 19:51:20.589800 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:20.589773 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4477e2-995c7ff66-8w9p8" Oct 30 19:51:20.661912 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:20.661869 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4477e2-995c7ff66-8w9p8"] Oct 30 19:51:20.684380 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:20.673609 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-4477e2\"" Oct 30 19:51:20.699015 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:20.698982 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-4477e2-dockercfg-bxpfr\"" Oct 30 19:51:20.839544 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:20.792727 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8q49\" (UniqueName: \"kubernetes.io/projected/56bef2d5-12bf-4ccb-8dea-f031595ddd96-kube-api-access-q8q49\") pod \"payload-processing-e2e-api-b-4477e2-995c7ff66-8w9p8\" (UID: \"56bef2d5-12bf-4ccb-8dea-f031595ddd96\") " pod="openshift-ingress/payload-processing-e2e-api-b-4477e2-995c7ff66-8w9p8" Oct 30 19:51:20.839544 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:20.792912 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56bef2d5-12bf-4ccb-8dea-f031595ddd96-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4477e2-995c7ff66-8w9p8\" (UID: \"56bef2d5-12bf-4ccb-8dea-f031595ddd96\") " pod="openshift-ingress/payload-processing-e2e-api-b-4477e2-995c7ff66-8w9p8" Oct 30 19:51:20.894258 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:20.894211 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56bef2d5-12bf-4ccb-8dea-f031595ddd96-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4477e2-995c7ff66-8w9p8\" (UID: \"56bef2d5-12bf-4ccb-8dea-f031595ddd96\") " pod="openshift-ingress/payload-processing-e2e-api-b-4477e2-995c7ff66-8w9p8" Oct 30 19:51:20.894670 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:20.894647 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8q49\" (UniqueName: \"kubernetes.io/projected/56bef2d5-12bf-4ccb-8dea-f031595ddd96-kube-api-access-q8q49\") pod \"payload-processing-e2e-api-b-4477e2-995c7ff66-8w9p8\" (UID: \"56bef2d5-12bf-4ccb-8dea-f031595ddd96\") " pod="openshift-ingress/payload-processing-e2e-api-b-4477e2-995c7ff66-8w9p8" Oct 30 19:51:20.896877 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:20.896792 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56bef2d5-12bf-4ccb-8dea-f031595ddd96-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4477e2-995c7ff66-8w9p8\" (UID: \"56bef2d5-12bf-4ccb-8dea-f031595ddd96\") " pod="openshift-ingress/payload-processing-e2e-api-b-4477e2-995c7ff66-8w9p8" Oct 30 19:51:20.900435 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:20.859734 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4477e2-55fc4bcff9-4x6qk"] Oct 30 19:51:20.904069 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:20.904052 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4477e2-55fc4bcff9-4x6qk"] Oct 30 19:51:20.904145 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:20.904133 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4477e2-55fc4bcff9-4x6qk" Oct 30 19:51:20.957735 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:20.957705 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8q49\" (UniqueName: \"kubernetes.io/projected/56bef2d5-12bf-4ccb-8dea-f031595ddd96-kube-api-access-q8q49\") pod \"payload-processing-e2e-api-b-4477e2-995c7ff66-8w9p8\" (UID: \"56bef2d5-12bf-4ccb-8dea-f031595ddd96\") " pod="openshift-ingress/payload-processing-e2e-api-b-4477e2-995c7ff66-8w9p8" Oct 30 19:51:20.996939 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:20.996906 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2hrx\" (UniqueName: \"kubernetes.io/projected/aa4437a1-8344-44c8-91a9-6ae667cbad68-kube-api-access-v2hrx\") pod \"payload-pre-processing-e2e-api-b-4477e2-55fc4bcff9-4x6qk\" (UID: \"aa4437a1-8344-44c8-91a9-6ae667cbad68\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4477e2-55fc4bcff9-4x6qk" Oct 30 19:51:20.997090 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:20.997052 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa4437a1-8344-44c8-91a9-6ae667cbad68-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4477e2-55fc4bcff9-4x6qk\" (UID: \"aa4437a1-8344-44c8-91a9-6ae667cbad68\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4477e2-55fc4bcff9-4x6qk" Oct 30 19:51:21.098494 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:21.098407 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2hrx\" (UniqueName: \"kubernetes.io/projected/aa4437a1-8344-44c8-91a9-6ae667cbad68-kube-api-access-v2hrx\") pod \"payload-pre-processing-e2e-api-b-4477e2-55fc4bcff9-4x6qk\" (UID: \"aa4437a1-8344-44c8-91a9-6ae667cbad68\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4477e2-55fc4bcff9-4x6qk" Oct 30 19:51:21.098494 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:21.098493 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa4437a1-8344-44c8-91a9-6ae667cbad68-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4477e2-55fc4bcff9-4x6qk\" (UID: \"aa4437a1-8344-44c8-91a9-6ae667cbad68\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4477e2-55fc4bcff9-4x6qk" Oct 30 19:51:21.099214 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:21.099185 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa4437a1-8344-44c8-91a9-6ae667cbad68-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4477e2-55fc4bcff9-4x6qk\" (UID: \"aa4437a1-8344-44c8-91a9-6ae667cbad68\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4477e2-55fc4bcff9-4x6qk" Oct 30 19:51:21.160962 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:21.160929 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2hrx\" (UniqueName: \"kubernetes.io/projected/aa4437a1-8344-44c8-91a9-6ae667cbad68-kube-api-access-v2hrx\") pod \"payload-pre-processing-e2e-api-b-4477e2-55fc4bcff9-4x6qk\" (UID: \"aa4437a1-8344-44c8-91a9-6ae667cbad68\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4477e2-55fc4bcff9-4x6qk" Oct 30 19:51:21.203047 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:21.203019 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4477e2-995c7ff66-8w9p8" Oct 30 19:51:21.219199 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:21.219174 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4477e2-55fc4bcff9-4x6qk" Oct 30 19:51:21.581575 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:21.581550 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4477e2-995c7ff66-8w9p8"] Oct 30 19:51:21.582065 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:51:21.582041 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56bef2d5_12bf_4ccb_8dea_f031595ddd96.slice/crio-47863e8f8299ca9d3dd8689873822b46f4e6484131339f6c50ed3da93871a552 WatchSource:0}: Error finding container 47863e8f8299ca9d3dd8689873822b46f4e6484131339f6c50ed3da93871a552: Status 404 returned error can't find the container with id 47863e8f8299ca9d3dd8689873822b46f4e6484131339f6c50ed3da93871a552 Oct 30 19:51:21.631751 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:21.631728 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4477e2-55fc4bcff9-4x6qk"] Oct 30 19:51:21.640603 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:51:21.640576 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa4437a1_8344_44c8_91a9_6ae667cbad68.slice/crio-446672aa3760aea0e910f00cf8c6ae7644dd98b2c2f5431c002b6509776f0b45 WatchSource:0}: Error finding container 446672aa3760aea0e910f00cf8c6ae7644dd98b2c2f5431c002b6509776f0b45: Status 404 returned error can't find the container with id 446672aa3760aea0e910f00cf8c6ae7644dd98b2c2f5431c002b6509776f0b45 Oct 30 19:51:22.199481 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:22.199437 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4477e2-995c7ff66-8w9p8" event={"ID":"56bef2d5-12bf-4ccb-8dea-f031595ddd96","Type":"ContainerStarted","Data":"6fd30c2b0d7cf92edc71af6e275c87a38f7a66ffccb94df682b0abbfeef6496a"} Oct 30 19:51:22.199685 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:22.199495 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4477e2-995c7ff66-8w9p8" event={"ID":"56bef2d5-12bf-4ccb-8dea-f031595ddd96","Type":"ContainerStarted","Data":"47863e8f8299ca9d3dd8689873822b46f4e6484131339f6c50ed3da93871a552"} Oct 30 19:51:22.199685 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:22.199622 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-4477e2-995c7ff66-8w9p8" Oct 30 19:51:22.201145 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:22.201120 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4477e2-55fc4bcff9-4x6qk" event={"ID":"aa4437a1-8344-44c8-91a9-6ae667cbad68","Type":"ContainerStarted","Data":"f8fbc5cb397c17b9c413487a01d8d502bcfda59ecb1cdadf89ec83b0074efb78"} Oct 30 19:51:22.201259 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:22.201164 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4477e2-55fc4bcff9-4x6qk" event={"ID":"aa4437a1-8344-44c8-91a9-6ae667cbad68","Type":"ContainerStarted","Data":"446672aa3760aea0e910f00cf8c6ae7644dd98b2c2f5431c002b6509776f0b45"} Oct 30 19:51:22.201259 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:22.201203 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4477e2-55fc4bcff9-4x6qk" Oct 30 19:51:22.319345 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:22.319293 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-4477e2-995c7ff66-8w9p8" podStartSLOduration=2.319278358 podStartE2EDuration="2.319278358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:22.317023766 +0000 UTC m=+1585.835339223" watchObservedRunningTime="2026-10-30 19:51:22.319278358 +0000 UTC m=+1585.837593815" Oct 30 19:51:22.445396 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:22.441912 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4477e2-55fc4bcff9-4x6qk" podStartSLOduration=2.441887663 podStartE2EDuration="2.441887663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:22.440381296 +0000 UTC m=+1585.958696751" watchObservedRunningTime="2026-10-30 19:51:22.441887663 +0000 UTC m=+1585.960203121" Oct 30 19:51:28.279052 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:28.278623 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-61d11a-7759448859-vzndl"] Oct 30 19:51:28.290241 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:28.290215 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-61d11a-7759448859-vzndl" Oct 30 19:51:28.313685 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:28.313659 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bb658ce-897b-46f3-8b91-97c611f6221d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-61d11a-7759448859-vzndl\" (UID: \"2bb658ce-897b-46f3-8b91-97c611f6221d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-61d11a-7759448859-vzndl" Oct 30 19:51:28.313811 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:28.313691 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdzqv\" (UniqueName: \"kubernetes.io/projected/2bb658ce-897b-46f3-8b91-97c611f6221d-kube-api-access-hdzqv\") pod \"payload-pre-processing-e2e-api-a-61d11a-7759448859-vzndl\" (UID: \"2bb658ce-897b-46f3-8b91-97c611f6221d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-61d11a-7759448859-vzndl" Oct 30 19:51:28.336218 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:28.336187 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-61d11a-7759448859-vzndl"] Oct 30 19:51:28.383485 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:28.383464 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-61d11a\"" Oct 30 19:51:28.383759 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:28.383741 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-61d11a-dockercfg-8sjcb\"" Oct 30 19:51:28.415516 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:28.415485 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bb658ce-897b-46f3-8b91-97c611f6221d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-61d11a-7759448859-vzndl\" (UID: \"2bb658ce-897b-46f3-8b91-97c611f6221d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-61d11a-7759448859-vzndl" Oct 30 19:51:28.415516 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:28.415517 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdzqv\" (UniqueName: \"kubernetes.io/projected/2bb658ce-897b-46f3-8b91-97c611f6221d-kube-api-access-hdzqv\") pod \"payload-pre-processing-e2e-api-a-61d11a-7759448859-vzndl\" (UID: \"2bb658ce-897b-46f3-8b91-97c611f6221d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-61d11a-7759448859-vzndl" Oct 30 19:51:28.416201 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:28.416182 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bb658ce-897b-46f3-8b91-97c611f6221d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-61d11a-7759448859-vzndl\" (UID: \"2bb658ce-897b-46f3-8b91-97c611f6221d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-61d11a-7759448859-vzndl" Oct 30 19:51:28.479103 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:28.479067 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdzqv\" (UniqueName: \"kubernetes.io/projected/2bb658ce-897b-46f3-8b91-97c611f6221d-kube-api-access-hdzqv\") pod \"payload-pre-processing-e2e-api-a-61d11a-7759448859-vzndl\" (UID: \"2bb658ce-897b-46f3-8b91-97c611f6221d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-61d11a-7759448859-vzndl" Oct 30 19:51:28.608559 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:28.608463 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-61d11a-7759448859-vzndl" Oct 30 19:51:28.989929 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:28.989901 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-61d11a-7759448859-vzndl"] Oct 30 19:51:28.990750 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:51:28.990720 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bb658ce_897b_46f3_8b91_97c611f6221d.slice/crio-01b6d6af7f1cb815ddb1a36ccd7976e36b98113ab38fe5aa4d1ab442bea56d6f WatchSource:0}: Error finding container 01b6d6af7f1cb815ddb1a36ccd7976e36b98113ab38fe5aa4d1ab442bea56d6f: Status 404 returned error can't find the container with id 01b6d6af7f1cb815ddb1a36ccd7976e36b98113ab38fe5aa4d1ab442bea56d6f Oct 30 19:51:29.317065 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:29.317030 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-61d11a-7759448859-vzndl" event={"ID":"2bb658ce-897b-46f3-8b91-97c611f6221d","Type":"ContainerStarted","Data":"da530990c6977328117161a7915bc7715d63d57b756d3beb29f63ce890195883"} Oct 30 19:51:29.317609 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:29.317091 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-61d11a-7759448859-vzndl" event={"ID":"2bb658ce-897b-46f3-8b91-97c611f6221d","Type":"ContainerStarted","Data":"01b6d6af7f1cb815ddb1a36ccd7976e36b98113ab38fe5aa4d1ab442bea56d6f"} Oct 30 19:51:29.317609 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:29.317255 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-61d11a-7759448859-vzndl" Oct 30 19:51:29.437379 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:29.437273 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-61d11a-7759448859-vzndl" podStartSLOduration=1.4372561130000001 podStartE2EDuration="1.437256113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:29.436077457 +0000 UTC m=+1592.954392969" watchObservedRunningTime="2026-10-30 19:51:29.437256113 +0000 UTC m=+1592.955571571" Oct 30 19:51:33.209470 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:33.209440 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4477e2-55fc4bcff9-4x6qk" Oct 30 19:51:33.209859 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:33.209508 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-4477e2-995c7ff66-8w9p8" Oct 30 19:51:40.326162 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:51:40.326126 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-61d11a-7759448859-vzndl" Oct 30 19:51:43.222763 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:51:43.222720 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:52:02.214506 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:02.214473 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4477e2-f97645c84-h7sqc"] Oct 30 19:52:02.219553 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:02.219533 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4477e2-f97645c84-h7sqc" Oct 30 19:52:02.329042 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:02.329003 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-4477e2-serving-cert\"" Oct 30 19:52:02.332195 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:02.332158 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4477e2-f97645c84-h7sqc"] Oct 30 19:52:02.347961 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:02.347922 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30db642e-9138-4699-8ffe-7f530f1e956e-maas-api-tls\") pod \"maas-api-e2e-api-b-4477e2-f97645c84-h7sqc\" (UID: \"30db642e-9138-4699-8ffe-7f530f1e956e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4477e2-f97645c84-h7sqc" Oct 30 19:52:02.348138 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:02.347975 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30db642e-9138-4699-8ffe-7f530f1e956e-metrics-certs\") pod \"maas-api-e2e-api-b-4477e2-f97645c84-h7sqc\" (UID: \"30db642e-9138-4699-8ffe-7f530f1e956e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4477e2-f97645c84-h7sqc" Oct 30 19:52:02.348138 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:02.348104 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rbcr\" (UniqueName: \"kubernetes.io/projected/30db642e-9138-4699-8ffe-7f530f1e956e-kube-api-access-9rbcr\") pod \"maas-api-e2e-api-b-4477e2-f97645c84-h7sqc\" (UID: \"30db642e-9138-4699-8ffe-7f530f1e956e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4477e2-f97645c84-h7sqc" Oct 30 19:52:02.348311 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:02.348289 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30db642e-9138-4699-8ffe-7f530f1e956e-openshift-service-ca\") pod \"maas-api-e2e-api-b-4477e2-f97645c84-h7sqc\" (UID: \"30db642e-9138-4699-8ffe-7f530f1e956e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4477e2-f97645c84-h7sqc" Oct 30 19:52:02.452667 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:02.452510 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30db642e-9138-4699-8ffe-7f530f1e956e-maas-api-tls\") pod \"maas-api-e2e-api-b-4477e2-f97645c84-h7sqc\" (UID: \"30db642e-9138-4699-8ffe-7f530f1e956e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4477e2-f97645c84-h7sqc" Oct 30 19:52:02.452667 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:02.452577 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30db642e-9138-4699-8ffe-7f530f1e956e-metrics-certs\") pod \"maas-api-e2e-api-b-4477e2-f97645c84-h7sqc\" (UID: \"30db642e-9138-4699-8ffe-7f530f1e956e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4477e2-f97645c84-h7sqc" Oct 30 19:52:02.452667 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:02.452645 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rbcr\" (UniqueName: \"kubernetes.io/projected/30db642e-9138-4699-8ffe-7f530f1e956e-kube-api-access-9rbcr\") pod \"maas-api-e2e-api-b-4477e2-f97645c84-h7sqc\" (UID: \"30db642e-9138-4699-8ffe-7f530f1e956e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4477e2-f97645c84-h7sqc" Oct 30 19:52:02.476694 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:52:02.452745 4292 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-4477e2-serving-cert: secret "maas-api-e2e-api-b-4477e2-serving-cert" not found Oct 30 19:52:02.476694 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:52:02.452850 4292 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30db642e-9138-4699-8ffe-7f530f1e956e-maas-api-tls podName:30db642e-9138-4699-8ffe-7f530f1e956e nodeName:}" failed. No retries permitted until 2026-10-30 19:52:02.952826616 +0000 UTC m=+1626.471142061 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/30db642e-9138-4699-8ffe-7f530f1e956e-maas-api-tls") pod "maas-api-e2e-api-b-4477e2-f97645c84-h7sqc" (UID: "30db642e-9138-4699-8ffe-7f530f1e956e") : secret "maas-api-e2e-api-b-4477e2-serving-cert" not found Oct 30 19:52:02.476694 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:02.453290 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30db642e-9138-4699-8ffe-7f530f1e956e-openshift-service-ca\") pod \"maas-api-e2e-api-b-4477e2-f97645c84-h7sqc\" (UID: \"30db642e-9138-4699-8ffe-7f530f1e956e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4477e2-f97645c84-h7sqc" Oct 30 19:52:02.476694 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:02.454630 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30db642e-9138-4699-8ffe-7f530f1e956e-openshift-service-ca\") pod \"maas-api-e2e-api-b-4477e2-f97645c84-h7sqc\" (UID: \"30db642e-9138-4699-8ffe-7f530f1e956e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4477e2-f97645c84-h7sqc" Oct 30 19:52:02.476694 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:02.476390 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30db642e-9138-4699-8ffe-7f530f1e956e-metrics-certs\") pod \"maas-api-e2e-api-b-4477e2-f97645c84-h7sqc\" (UID: \"30db642e-9138-4699-8ffe-7f530f1e956e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4477e2-f97645c84-h7sqc" Oct 30 19:52:02.535615 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:02.535578 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rbcr\" (UniqueName: \"kubernetes.io/projected/30db642e-9138-4699-8ffe-7f530f1e956e-kube-api-access-9rbcr\") pod \"maas-api-e2e-api-b-4477e2-f97645c84-h7sqc\" (UID: \"30db642e-9138-4699-8ffe-7f530f1e956e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4477e2-f97645c84-h7sqc" Oct 30 19:52:02.967765 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:02.967731 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30db642e-9138-4699-8ffe-7f530f1e956e-maas-api-tls\") pod \"maas-api-e2e-api-b-4477e2-f97645c84-h7sqc\" (UID: \"30db642e-9138-4699-8ffe-7f530f1e956e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4477e2-f97645c84-h7sqc" Oct 30 19:52:02.970099 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:02.970077 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30db642e-9138-4699-8ffe-7f530f1e956e-maas-api-tls\") pod \"maas-api-e2e-api-b-4477e2-f97645c84-h7sqc\" (UID: \"30db642e-9138-4699-8ffe-7f530f1e956e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4477e2-f97645c84-h7sqc" Oct 30 19:52:03.132804 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:03.132756 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4477e2-f97645c84-h7sqc" Oct 30 19:52:03.510428 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:03.510401 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4477e2-f97645c84-h7sqc"] Oct 30 19:52:03.511442 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:52:03.511410 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30db642e_9138_4699_8ffe_7f530f1e956e.slice/crio-8e32d8f982cf54aea279c7233c839f973fd5047d4f13732f23392c099ad13465 WatchSource:0}: Error finding container 8e32d8f982cf54aea279c7233c839f973fd5047d4f13732f23392c099ad13465: Status 404 returned error can't find the container with id 8e32d8f982cf54aea279c7233c839f973fd5047d4f13732f23392c099ad13465 Oct 30 19:52:03.512948 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:03.512927 4292 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:52:03.532622 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:03.532588 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4477e2-f97645c84-h7sqc" event={"ID":"30db642e-9138-4699-8ffe-7f530f1e956e","Type":"ContainerStarted","Data":"8e32d8f982cf54aea279c7233c839f973fd5047d4f13732f23392c099ad13465"} Oct 30 19:52:04.541217 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:04.541175 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4477e2-f97645c84-h7sqc" event={"ID":"30db642e-9138-4699-8ffe-7f530f1e956e","Type":"ContainerStarted","Data":"b41b10ff578fd659bf128a0d2ebadfb34e3a52533d3836a0f06a8dcdac36d529"} Oct 30 19:52:04.541660 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:04.541280 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4477e2-f97645c84-h7sqc" Oct 30 19:52:04.667640 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:04.667521 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4477e2-f97645c84-h7sqc" podStartSLOduration=1.8616914549999999 podStartE2EDuration="2.667501375s" totalImagesPullingTime="805.80992ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:04.664964606 +0000 UTC m=+1628.183280064" watchObservedRunningTime="2026-10-30 19:52:04.667501375 +0000 UTC m=+1628.185816833" Oct 30 19:52:10.558096 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:10.558062 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4477e2-f97645c84-h7sqc" Oct 30 19:52:12.544308 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:12.544270 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-61d11a-555b777f7d-6zlhz"] Oct 30 19:52:12.548685 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:12.548656 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61d11a-555b777f7d-6zlhz" Oct 30 19:52:12.602749 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:12.602716 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-61d11a-serving-cert\"" Oct 30 19:52:12.605850 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:12.605817 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-61d11a-555b777f7d-6zlhz"] Oct 30 19:52:12.617748 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:12.617713 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft2jf\" (UniqueName: \"kubernetes.io/projected/fb0e159a-7596-4936-989d-0cc0cdc952af-kube-api-access-ft2jf\") pod \"maas-api-e2e-api-a-61d11a-555b777f7d-6zlhz\" (UID: \"fb0e159a-7596-4936-989d-0cc0cdc952af\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61d11a-555b777f7d-6zlhz" Oct 30 19:52:12.617973 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:12.617943 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb0e159a-7596-4936-989d-0cc0cdc952af-openshift-service-ca\") pod \"maas-api-e2e-api-a-61d11a-555b777f7d-6zlhz\" (UID: \"fb0e159a-7596-4936-989d-0cc0cdc952af\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61d11a-555b777f7d-6zlhz" Oct 30 19:52:12.618092 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:12.618076 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb0e159a-7596-4936-989d-0cc0cdc952af-metrics-certs\") pod \"maas-api-e2e-api-a-61d11a-555b777f7d-6zlhz\" (UID: \"fb0e159a-7596-4936-989d-0cc0cdc952af\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61d11a-555b777f7d-6zlhz" Oct 30 19:52:12.618129 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:12.618116 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb0e159a-7596-4936-989d-0cc0cdc952af-maas-api-tls\") pod \"maas-api-e2e-api-a-61d11a-555b777f7d-6zlhz\" (UID: \"fb0e159a-7596-4936-989d-0cc0cdc952af\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61d11a-555b777f7d-6zlhz" Oct 30 19:52:12.719056 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:12.719025 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb0e159a-7596-4936-989d-0cc0cdc952af-metrics-certs\") pod \"maas-api-e2e-api-a-61d11a-555b777f7d-6zlhz\" (UID: \"fb0e159a-7596-4936-989d-0cc0cdc952af\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61d11a-555b777f7d-6zlhz" Oct 30 19:52:12.719245 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:12.719071 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb0e159a-7596-4936-989d-0cc0cdc952af-maas-api-tls\") pod \"maas-api-e2e-api-a-61d11a-555b777f7d-6zlhz\" (UID: \"fb0e159a-7596-4936-989d-0cc0cdc952af\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61d11a-555b777f7d-6zlhz" Oct 30 19:52:12.719245 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:12.719116 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft2jf\" (UniqueName: \"kubernetes.io/projected/fb0e159a-7596-4936-989d-0cc0cdc952af-kube-api-access-ft2jf\") pod \"maas-api-e2e-api-a-61d11a-555b777f7d-6zlhz\" (UID: \"fb0e159a-7596-4936-989d-0cc0cdc952af\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61d11a-555b777f7d-6zlhz" Oct 30 19:52:12.719245 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:52:12.719225 4292 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-61d11a-serving-cert: secret "maas-api-e2e-api-a-61d11a-serving-cert" not found Oct 30 19:52:12.719437 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:52:12.719294 4292 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb0e159a-7596-4936-989d-0cc0cdc952af-maas-api-tls podName:fb0e159a-7596-4936-989d-0cc0cdc952af nodeName:}" failed. No retries permitted until 2026-10-30 19:52:13.219273627 +0000 UTC m=+1636.737589066 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fb0e159a-7596-4936-989d-0cc0cdc952af-maas-api-tls") pod "maas-api-e2e-api-a-61d11a-555b777f7d-6zlhz" (UID: "fb0e159a-7596-4936-989d-0cc0cdc952af") : secret "maas-api-e2e-api-a-61d11a-serving-cert" not found Oct 30 19:52:12.719437 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:12.719322 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb0e159a-7596-4936-989d-0cc0cdc952af-openshift-service-ca\") pod \"maas-api-e2e-api-a-61d11a-555b777f7d-6zlhz\" (UID: \"fb0e159a-7596-4936-989d-0cc0cdc952af\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61d11a-555b777f7d-6zlhz" Oct 30 19:52:12.720019 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:12.719992 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb0e159a-7596-4936-989d-0cc0cdc952af-openshift-service-ca\") pod \"maas-api-e2e-api-a-61d11a-555b777f7d-6zlhz\" (UID: \"fb0e159a-7596-4936-989d-0cc0cdc952af\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61d11a-555b777f7d-6zlhz" Oct 30 19:52:12.721524 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:12.721501 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb0e159a-7596-4936-989d-0cc0cdc952af-metrics-certs\") pod \"maas-api-e2e-api-a-61d11a-555b777f7d-6zlhz\" (UID: \"fb0e159a-7596-4936-989d-0cc0cdc952af\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61d11a-555b777f7d-6zlhz" Oct 30 19:52:12.781942 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:12.781909 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft2jf\" (UniqueName: \"kubernetes.io/projected/fb0e159a-7596-4936-989d-0cc0cdc952af-kube-api-access-ft2jf\") pod \"maas-api-e2e-api-a-61d11a-555b777f7d-6zlhz\" (UID: \"fb0e159a-7596-4936-989d-0cc0cdc952af\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61d11a-555b777f7d-6zlhz" Oct 30 19:52:13.222333 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:52:13.222291 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:52:13.229213 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:13.229181 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb0e159a-7596-4936-989d-0cc0cdc952af-maas-api-tls\") pod \"maas-api-e2e-api-a-61d11a-555b777f7d-6zlhz\" (UID: \"fb0e159a-7596-4936-989d-0cc0cdc952af\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61d11a-555b777f7d-6zlhz" Oct 30 19:52:13.231714 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:13.231685 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb0e159a-7596-4936-989d-0cc0cdc952af-maas-api-tls\") pod \"maas-api-e2e-api-a-61d11a-555b777f7d-6zlhz\" (UID: \"fb0e159a-7596-4936-989d-0cc0cdc952af\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61d11a-555b777f7d-6zlhz" Oct 30 19:52:13.463457 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:13.463423 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61d11a-555b777f7d-6zlhz" Oct 30 19:52:13.845441 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:13.845412 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-61d11a-555b777f7d-6zlhz"] Oct 30 19:52:13.846230 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:52:13.846201 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb0e159a_7596_4936_989d_0cc0cdc952af.slice/crio-5c325ddf05e9f2da44b90f43afc75b6b8ada101b6f99f22febcfe888e5ae0a93 WatchSource:0}: Error finding container 5c325ddf05e9f2da44b90f43afc75b6b8ada101b6f99f22febcfe888e5ae0a93: Status 404 returned error can't find the container with id 5c325ddf05e9f2da44b90f43afc75b6b8ada101b6f99f22febcfe888e5ae0a93 Oct 30 19:52:14.603933 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:14.603896 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61d11a-555b777f7d-6zlhz" event={"ID":"fb0e159a-7596-4936-989d-0cc0cdc952af","Type":"ContainerStarted","Data":"5c325ddf05e9f2da44b90f43afc75b6b8ada101b6f99f22febcfe888e5ae0a93"} Oct 30 19:52:15.610839 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:15.610805 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61d11a-555b777f7d-6zlhz" event={"ID":"fb0e159a-7596-4936-989d-0cc0cdc952af","Type":"ContainerStarted","Data":"925c6b8554f72f55e93bb3528b52c7d9fd079e56d878957748a01f480422864a"} Oct 30 19:52:15.611255 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:15.610988 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61d11a-555b777f7d-6zlhz" Oct 30 19:52:15.727421 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:15.727386 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft"] Oct 30 19:52:15.731321 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:15.731302 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft" Oct 30 19:52:15.767242 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:15.767205 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caf34ded-76a9-4af9-9b38-bd5437157111-workload-certs\") pod \"e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft\" (UID: \"caf34ded-76a9-4af9-9b38-bd5437157111\") " pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft" Oct 30 19:52:15.767242 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:15.767246 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caf34ded-76a9-4af9-9b38-bd5437157111-istio-envoy\") pod \"e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft\" (UID: \"caf34ded-76a9-4af9-9b38-bd5437157111\") " pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft" Oct 30 19:52:15.767495 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:15.767334 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caf34ded-76a9-4af9-9b38-bd5437157111-istio-token\") pod \"e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft\" (UID: \"caf34ded-76a9-4af9-9b38-bd5437157111\") " pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft" Oct 30 19:52:15.767543 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:15.767501 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caf34ded-76a9-4af9-9b38-bd5437157111-istiod-ca-cert\") pod \"e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft\" (UID: \"caf34ded-76a9-4af9-9b38-bd5437157111\") " pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft" Oct 30 19:52:15.767585 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:15.767560 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caf34ded-76a9-4af9-9b38-bd5437157111-istio-data\") pod \"e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft\" (UID: \"caf34ded-76a9-4af9-9b38-bd5437157111\") " pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft" Oct 30 19:52:15.767622 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:15.767589 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf34ded-76a9-4af9-9b38-bd5437157111-credential-socket\") pod \"e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft\" (UID: \"caf34ded-76a9-4af9-9b38-bd5437157111\") " pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft" Oct 30 19:52:15.767664 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:15.767644 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfhtj\" (UniqueName: \"kubernetes.io/projected/caf34ded-76a9-4af9-9b38-bd5437157111-kube-api-access-cfhtj\") pod \"e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft\" (UID: \"caf34ded-76a9-4af9-9b38-bd5437157111\") " pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft" Oct 30 19:52:15.767701 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:15.767675 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf34ded-76a9-4af9-9b38-bd5437157111-workload-socket\") pod \"e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft\" (UID: \"caf34ded-76a9-4af9-9b38-bd5437157111\") " pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft" Oct 30 19:52:15.767744 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:15.767702 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caf34ded-76a9-4af9-9b38-bd5437157111-istio-podinfo\") pod \"e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft\" (UID: \"caf34ded-76a9-4af9-9b38-bd5437157111\") " pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft" Oct 30 19:52:15.775629 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:15.775586 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61d11a-555b777f7d-6zlhz" podStartSLOduration=2.957201917 podStartE2EDuration="3.775576067s" totalImagesPullingTime="818.37415ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:15.734019785 +0000 UTC m=+1639.252335241" watchObservedRunningTime="2026-10-30 19:52:15.775576067 +0000 UTC m=+1639.293891550" Oct 30 19:52:15.820147 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:15.820111 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft"] Oct 30 19:52:15.866881 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:15.866792 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-cf283d-openshift-default-dockercfg-l2hvl\"" Oct 30 19:52:15.869415 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:15.869390 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfhtj\" (UniqueName: \"kubernetes.io/projected/caf34ded-76a9-4af9-9b38-bd5437157111-kube-api-access-cfhtj\") pod \"e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft\" (UID: \"caf34ded-76a9-4af9-9b38-bd5437157111\") " pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft" Oct 30 19:52:15.869530 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:15.869425 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf34ded-76a9-4af9-9b38-bd5437157111-workload-socket\") pod \"e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft\" (UID: \"caf34ded-76a9-4af9-9b38-bd5437157111\") " pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft" Oct 30 19:52:15.869530 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:15.869456 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caf34ded-76a9-4af9-9b38-bd5437157111-istio-podinfo\") pod \"e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft\" (UID: \"caf34ded-76a9-4af9-9b38-bd5437157111\") " pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft" Oct 30 19:52:15.869638 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:15.869531 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caf34ded-76a9-4af9-9b38-bd5437157111-workload-certs\") pod \"e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft\" (UID: \"caf34ded-76a9-4af9-9b38-bd5437157111\") " pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft" Oct 30 19:52:15.869638 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:15.869565 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caf34ded-76a9-4af9-9b38-bd5437157111-istio-envoy\") pod \"e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft\" (UID: \"caf34ded-76a9-4af9-9b38-bd5437157111\") " pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft" Oct 30 19:52:15.869638 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:15.869608 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caf34ded-76a9-4af9-9b38-bd5437157111-istio-token\") pod \"e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft\" (UID: \"caf34ded-76a9-4af9-9b38-bd5437157111\") " pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft" Oct 30 19:52:15.869820 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:15.869798 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caf34ded-76a9-4af9-9b38-bd5437157111-istiod-ca-cert\") pod \"e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft\" (UID: \"caf34ded-76a9-4af9-9b38-bd5437157111\") " pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft" Oct 30 19:52:15.869874 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:15.869839 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caf34ded-76a9-4af9-9b38-bd5437157111-istio-data\") pod \"e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft\" (UID: \"caf34ded-76a9-4af9-9b38-bd5437157111\") " pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft" Oct 30 19:52:15.869874 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:15.869854 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf34ded-76a9-4af9-9b38-bd5437157111-workload-socket\") pod \"e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft\" (UID: \"caf34ded-76a9-4af9-9b38-bd5437157111\") " pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft" Oct 30 19:52:15.869874 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:15.869869 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf34ded-76a9-4af9-9b38-bd5437157111-credential-socket\") pod \"e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft\" (UID: \"caf34ded-76a9-4af9-9b38-bd5437157111\") " pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft" Oct 30 19:52:15.870034 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:15.869932 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caf34ded-76a9-4af9-9b38-bd5437157111-workload-certs\") pod \"e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft\" (UID: \"caf34ded-76a9-4af9-9b38-bd5437157111\") " pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft" Oct 30 19:52:15.870135 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:15.870114 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf34ded-76a9-4af9-9b38-bd5437157111-credential-socket\") pod \"e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft\" (UID: \"caf34ded-76a9-4af9-9b38-bd5437157111\") " pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft" Oct 30 19:52:15.870197 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:15.870175 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caf34ded-76a9-4af9-9b38-bd5437157111-istio-data\") pod \"e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft\" (UID: \"caf34ded-76a9-4af9-9b38-bd5437157111\") " pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft" Oct 30 19:52:15.870739 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:15.870706 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caf34ded-76a9-4af9-9b38-bd5437157111-istiod-ca-cert\") pod \"e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft\" (UID: \"caf34ded-76a9-4af9-9b38-bd5437157111\") " pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft" Oct 30 19:52:15.872009 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:15.871985 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caf34ded-76a9-4af9-9b38-bd5437157111-istio-envoy\") pod \"e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft\" (UID: \"caf34ded-76a9-4af9-9b38-bd5437157111\") " pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft" Oct 30 19:52:15.872119 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:15.872088 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caf34ded-76a9-4af9-9b38-bd5437157111-istio-podinfo\") pod \"e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft\" (UID: \"caf34ded-76a9-4af9-9b38-bd5437157111\") " pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft" Oct 30 19:52:15.932044 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:15.932007 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caf34ded-76a9-4af9-9b38-bd5437157111-istio-token\") pod \"e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft\" (UID: \"caf34ded-76a9-4af9-9b38-bd5437157111\") " pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft" Oct 30 19:52:15.932200 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:15.932058 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfhtj\" (UniqueName: \"kubernetes.io/projected/caf34ded-76a9-4af9-9b38-bd5437157111-kube-api-access-cfhtj\") pod \"e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft\" (UID: \"caf34ded-76a9-4af9-9b38-bd5437157111\") " pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft" Oct 30 19:52:16.055899 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:16.054653 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft" Oct 30 19:52:16.587510 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:16.587485 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft"] Oct 30 19:52:16.587735 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:52:16.587713 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaf34ded_76a9_4af9_9b38_bd5437157111.slice/crio-70a3d7a05c49c99352b52ed185e3ed907c5b42108be37d56c946bee84609309f WatchSource:0}: Error finding container 70a3d7a05c49c99352b52ed185e3ed907c5b42108be37d56c946bee84609309f: Status 404 returned error can't find the container with id 70a3d7a05c49c99352b52ed185e3ed907c5b42108be37d56c946bee84609309f Oct 30 19:52:16.590194 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:16.590163 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:52:16.590278 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:16.590225 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:52:16.590278 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:16.590254 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:52:16.617527 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:16.617492 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft" event={"ID":"caf34ded-76a9-4af9-9b38-bd5437157111","Type":"ContainerStarted","Data":"70a3d7a05c49c99352b52ed185e3ed907c5b42108be37d56c946bee84609309f"} Oct 30 19:52:17.917230 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:17.917191 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft" event={"ID":"caf34ded-76a9-4af9-9b38-bd5437157111","Type":"ContainerStarted","Data":"7987bd9efcbc925efb40e00456f4d268632ce18de216e258866f2c03687a39e3"} Oct 30 19:52:18.056774 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:18.056736 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft" Oct 30 19:52:18.059292 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:18.059255 4292 patch_prober.go:29] interesting pod/e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 19:52:18.059613 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:18.059562 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft" podUID="caf34ded-76a9-4af9-9b38-bd5437157111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 19:52:18.115682 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:18.115616 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft" podStartSLOduration=3.115593703 podStartE2EDuration="3.115593703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:18.111888056 +0000 UTC m=+1641.630203514" watchObservedRunningTime="2026-10-30 19:52:18.115593703 +0000 UTC m=+1641.633909161" Oct 30 19:52:19.059373 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:19.057124 4292 patch_prober.go:29] interesting pod/e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 19:52:19.059373 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:19.057188 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft" podUID="caf34ded-76a9-4af9-9b38-bd5437157111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 19:52:20.056839 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:20.056796 4292 patch_prober.go:29] interesting pod/e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 19:52:20.057131 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:20.057100 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft" podUID="caf34ded-76a9-4af9-9b38-bd5437157111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 19:52:21.071379 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:21.064984 4292 patch_prober.go:29] interesting pod/e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 19:52:21.071379 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:21.065074 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft" podUID="caf34ded-76a9-4af9-9b38-bd5437157111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 19:52:21.623902 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:21.623869 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61d11a-555b777f7d-6zlhz" Oct 30 19:52:22.056819 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:22.056776 4292 patch_prober.go:29] interesting pod/e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 19:52:22.057117 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:22.057085 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft" podUID="caf34ded-76a9-4af9-9b38-bd5437157111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 19:52:22.378551 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:22.378426 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c"] Oct 30 19:52:22.387023 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:22.386993 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c" Oct 30 19:52:22.401591 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:22.401551 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02e3cbb1-bde4-4a15-b543-db2b4b820f99-istio-podinfo\") pod \"e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c\" (UID: \"02e3cbb1-bde4-4a15-b543-db2b4b820f99\") " pod="openshift-ingress/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c" Oct 30 19:52:22.402009 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:22.401983 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02e3cbb1-bde4-4a15-b543-db2b4b820f99-workload-certs\") pod \"e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c\" (UID: \"02e3cbb1-bde4-4a15-b543-db2b4b820f99\") " pod="openshift-ingress/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c" Oct 30 19:52:22.402199 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:22.402182 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02e3cbb1-bde4-4a15-b543-db2b4b820f99-istio-token\") pod \"e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c\" (UID: \"02e3cbb1-bde4-4a15-b543-db2b4b820f99\") " pod="openshift-ingress/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c" Oct 30 19:52:22.402436 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:22.402404 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02e3cbb1-bde4-4a15-b543-db2b4b820f99-credential-socket\") pod \"e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c\" (UID: \"02e3cbb1-bde4-4a15-b543-db2b4b820f99\") " pod="openshift-ingress/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c" Oct 30 19:52:22.402621 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:22.402605 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02e3cbb1-bde4-4a15-b543-db2b4b820f99-istio-data\") pod \"e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c\" (UID: \"02e3cbb1-bde4-4a15-b543-db2b4b820f99\") " pod="openshift-ingress/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c" Oct 30 19:52:22.402764 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:22.402751 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02e3cbb1-bde4-4a15-b543-db2b4b820f99-workload-socket\") pod \"e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c\" (UID: \"02e3cbb1-bde4-4a15-b543-db2b4b820f99\") " pod="openshift-ingress/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c" Oct 30 19:52:22.402959 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:22.402943 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcwz6\" (UniqueName: \"kubernetes.io/projected/02e3cbb1-bde4-4a15-b543-db2b4b820f99-kube-api-access-bcwz6\") pod \"e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c\" (UID: \"02e3cbb1-bde4-4a15-b543-db2b4b820f99\") " pod="openshift-ingress/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c" Oct 30 19:52:22.403141 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:22.403117 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02e3cbb1-bde4-4a15-b543-db2b4b820f99-istiod-ca-cert\") pod \"e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c\" (UID: \"02e3cbb1-bde4-4a15-b543-db2b4b820f99\") " pod="openshift-ingress/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c" Oct 30 19:52:22.403308 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:22.403293 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02e3cbb1-bde4-4a15-b543-db2b4b820f99-istio-envoy\") pod \"e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c\" (UID: \"02e3cbb1-bde4-4a15-b543-db2b4b820f99\") " pod="openshift-ingress/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c" Oct 30 19:52:22.447021 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:22.446990 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-69932e-openshift-default-dockercfg-crrps\"" Oct 30 19:52:22.542376 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:22.512151 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02e3cbb1-bde4-4a15-b543-db2b4b820f99-istio-podinfo\") pod \"e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c\" (UID: \"02e3cbb1-bde4-4a15-b543-db2b4b820f99\") " pod="openshift-ingress/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c" Oct 30 19:52:22.542376 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:22.512273 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02e3cbb1-bde4-4a15-b543-db2b4b820f99-workload-certs\") pod \"e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c\" (UID: \"02e3cbb1-bde4-4a15-b543-db2b4b820f99\") " pod="openshift-ingress/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c" Oct 30 19:52:22.542376 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:22.512301 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02e3cbb1-bde4-4a15-b543-db2b4b820f99-istio-token\") pod \"e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c\" (UID: \"02e3cbb1-bde4-4a15-b543-db2b4b820f99\") " pod="openshift-ingress/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c" Oct 30 19:52:22.542376 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:22.512367 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02e3cbb1-bde4-4a15-b543-db2b4b820f99-credential-socket\") pod \"e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c\" (UID: \"02e3cbb1-bde4-4a15-b543-db2b4b820f99\") " pod="openshift-ingress/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c" Oct 30 19:52:22.542376 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:22.512407 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02e3cbb1-bde4-4a15-b543-db2b4b820f99-istio-data\") pod \"e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c\" (UID: \"02e3cbb1-bde4-4a15-b543-db2b4b820f99\") " pod="openshift-ingress/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c" Oct 30 19:52:22.542376 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:22.512435 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02e3cbb1-bde4-4a15-b543-db2b4b820f99-workload-socket\") pod \"e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c\" (UID: \"02e3cbb1-bde4-4a15-b543-db2b4b820f99\") " pod="openshift-ingress/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c" Oct 30 19:52:22.542376 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:22.512511 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcwz6\" (UniqueName: \"kubernetes.io/projected/02e3cbb1-bde4-4a15-b543-db2b4b820f99-kube-api-access-bcwz6\") pod \"e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c\" (UID: \"02e3cbb1-bde4-4a15-b543-db2b4b820f99\") " pod="openshift-ingress/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c" Oct 30 19:52:22.542376 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:22.512553 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02e3cbb1-bde4-4a15-b543-db2b4b820f99-istiod-ca-cert\") pod \"e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c\" (UID: \"02e3cbb1-bde4-4a15-b543-db2b4b820f99\") " pod="openshift-ingress/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c" Oct 30 19:52:22.542376 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:22.512584 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02e3cbb1-bde4-4a15-b543-db2b4b820f99-istio-envoy\") pod \"e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c\" (UID: \"02e3cbb1-bde4-4a15-b543-db2b4b820f99\") " pod="openshift-ingress/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c" Oct 30 19:52:22.567255 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:22.566762 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02e3cbb1-bde4-4a15-b543-db2b4b820f99-istio-envoy\") pod \"e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c\" (UID: \"02e3cbb1-bde4-4a15-b543-db2b4b820f99\") " pod="openshift-ingress/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c" Oct 30 19:52:22.570172 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:22.567647 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02e3cbb1-bde4-4a15-b543-db2b4b820f99-workload-certs\") pod \"e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c\" (UID: \"02e3cbb1-bde4-4a15-b543-db2b4b820f99\") " pod="openshift-ingress/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c" Oct 30 19:52:22.570172 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:22.568194 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02e3cbb1-bde4-4a15-b543-db2b4b820f99-credential-socket\") pod \"e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c\" (UID: \"02e3cbb1-bde4-4a15-b543-db2b4b820f99\") " pod="openshift-ingress/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c" Oct 30 19:52:22.570172 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:22.568700 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02e3cbb1-bde4-4a15-b543-db2b4b820f99-workload-socket\") pod \"e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c\" (UID: \"02e3cbb1-bde4-4a15-b543-db2b4b820f99\") " pod="openshift-ingress/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c" Oct 30 19:52:22.586474 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:22.584799 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02e3cbb1-bde4-4a15-b543-db2b4b820f99-istiod-ca-cert\") pod \"e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c\" (UID: \"02e3cbb1-bde4-4a15-b543-db2b4b820f99\") " pod="openshift-ingress/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c" Oct 30 19:52:22.626378 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:22.626263 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02e3cbb1-bde4-4a15-b543-db2b4b820f99-istio-data\") pod \"e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c\" (UID: \"02e3cbb1-bde4-4a15-b543-db2b4b820f99\") " pod="openshift-ingress/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c" Oct 30 19:52:22.684441 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:22.683634 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcwz6\" (UniqueName: \"kubernetes.io/projected/02e3cbb1-bde4-4a15-b543-db2b4b820f99-kube-api-access-bcwz6\") pod \"e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c\" (UID: \"02e3cbb1-bde4-4a15-b543-db2b4b820f99\") " pod="openshift-ingress/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c" Oct 30 19:52:22.804493 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:22.804389 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02e3cbb1-bde4-4a15-b543-db2b4b820f99-istio-token\") pod \"e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c\" (UID: \"02e3cbb1-bde4-4a15-b543-db2b4b820f99\") " pod="openshift-ingress/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c" Oct 30 19:52:22.806233 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:22.806203 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c"] Oct 30 19:52:22.849101 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:22.849065 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02e3cbb1-bde4-4a15-b543-db2b4b820f99-istio-podinfo\") pod \"e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c\" (UID: \"02e3cbb1-bde4-4a15-b543-db2b4b820f99\") " pod="openshift-ingress/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c" Oct 30 19:52:23.065879 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:23.065836 4292 patch_prober.go:29] interesting pod/e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 19:52:23.066105 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:23.065932 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft" podUID="caf34ded-76a9-4af9-9b38-bd5437157111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 19:52:23.083056 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:23.083015 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c" Oct 30 19:52:23.559003 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:52:23.558969 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02e3cbb1_bde4_4a15_b543_db2b4b820f99.slice/crio-a5ece29fba64bffd1176e01abe0697fa09ad023a455b83c0535063fc5ac45e28 WatchSource:0}: Error finding container a5ece29fba64bffd1176e01abe0697fa09ad023a455b83c0535063fc5ac45e28: Status 404 returned error can't find the container with id a5ece29fba64bffd1176e01abe0697fa09ad023a455b83c0535063fc5ac45e28 Oct 30 19:52:23.559425 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:23.559063 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c"] Oct 30 19:52:23.562171 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:23.562136 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:52:23.562254 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:23.562224 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:52:23.562299 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:23.562269 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:52:23.956894 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:23.956777 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c" event={"ID":"02e3cbb1-bde4-4a15-b543-db2b4b820f99","Type":"ContainerStarted","Data":"28ea1d303ec7b4268abcd0b79525c9c6173ea784f25e5c8364afd72e9e299319"} Oct 30 19:52:23.956894 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:23.956856 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c" event={"ID":"02e3cbb1-bde4-4a15-b543-db2b4b820f99","Type":"ContainerStarted","Data":"a5ece29fba64bffd1176e01abe0697fa09ad023a455b83c0535063fc5ac45e28"} Oct 30 19:52:24.061206 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:24.061158 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft" Oct 30 19:52:24.061593 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:24.061561 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft" Oct 30 19:52:24.062341 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:24.062322 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft" Oct 30 19:52:24.081871 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:24.081819 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c" podStartSLOduration=2.081804658 podStartE2EDuration="2.081804658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:24.079205295 +0000 UTC m=+1647.597520791" watchObservedRunningTime="2026-10-30 19:52:24.081804658 +0000 UTC m=+1647.600120493" Oct 30 19:52:24.084146 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:24.084121 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c" Oct 30 19:52:24.085662 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:24.085637 4292 patch_prober.go:29] interesting pod/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 19:52:24.085767 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:24.085681 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c" podUID="02e3cbb1-bde4-4a15-b543-db2b4b820f99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 19:52:25.088522 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:25.088493 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c" Oct 30 19:52:26.010144 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:26.010109 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c" Oct 30 19:52:26.084946 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:26.084907 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c" Oct 30 19:52:31.876376 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:31.871185 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-69932e-6d756dcf5d-c8w5p"] Oct 30 19:52:31.898372 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:31.894165 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-69932e-6d756dcf5d-c8w5p" Oct 30 19:52:31.947389 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:31.927569 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-69932e-6d756dcf5d-c8w5p"] Oct 30 19:52:31.977382 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:31.965177 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1f916af-267b-4057-a8aa-eef309537220-plugins-config-volume\") pod \"payload-processing-e2e-api-a-69932e-6d756dcf5d-c8w5p\" (UID: \"a1f916af-267b-4057-a8aa-eef309537220\") " pod="openshift-ingress/payload-processing-e2e-api-a-69932e-6d756dcf5d-c8w5p" Oct 30 19:52:31.985384 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:31.977839 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb955\" (UniqueName: \"kubernetes.io/projected/a1f916af-267b-4057-a8aa-eef309537220-kube-api-access-nb955\") pod \"payload-processing-e2e-api-a-69932e-6d756dcf5d-c8w5p\" (UID: \"a1f916af-267b-4057-a8aa-eef309537220\") " pod="openshift-ingress/payload-processing-e2e-api-a-69932e-6d756dcf5d-c8w5p" Oct 30 19:52:32.055899 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:32.055868 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-69932e-dockercfg-rqbnp\"" Oct 30 19:52:32.056402 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:32.056377 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-69932e\"" Oct 30 19:52:32.076800 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:32.076765 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-69932e-774bbd5475-gvztq"] Oct 30 19:52:32.083129 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:32.083095 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-69932e-774bbd5475-gvztq" Oct 30 19:52:32.087260 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:32.087222 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb955\" (UniqueName: \"kubernetes.io/projected/a1f916af-267b-4057-a8aa-eef309537220-kube-api-access-nb955\") pod \"payload-processing-e2e-api-a-69932e-6d756dcf5d-c8w5p\" (UID: \"a1f916af-267b-4057-a8aa-eef309537220\") " pod="openshift-ingress/payload-processing-e2e-api-a-69932e-6d756dcf5d-c8w5p" Oct 30 19:52:32.088105 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:32.087528 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1f916af-267b-4057-a8aa-eef309537220-plugins-config-volume\") pod \"payload-processing-e2e-api-a-69932e-6d756dcf5d-c8w5p\" (UID: \"a1f916af-267b-4057-a8aa-eef309537220\") " pod="openshift-ingress/payload-processing-e2e-api-a-69932e-6d756dcf5d-c8w5p" Oct 30 19:52:32.088703 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:32.088648 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1f916af-267b-4057-a8aa-eef309537220-plugins-config-volume\") pod \"payload-processing-e2e-api-a-69932e-6d756dcf5d-c8w5p\" (UID: \"a1f916af-267b-4057-a8aa-eef309537220\") " pod="openshift-ingress/payload-processing-e2e-api-a-69932e-6d756dcf5d-c8w5p" Oct 30 19:52:32.101396 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:32.101331 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-69932e-774bbd5475-gvztq"] Oct 30 19:52:32.151583 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:32.151500 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb955\" (UniqueName: \"kubernetes.io/projected/a1f916af-267b-4057-a8aa-eef309537220-kube-api-access-nb955\") pod \"payload-processing-e2e-api-a-69932e-6d756dcf5d-c8w5p\" (UID: \"a1f916af-267b-4057-a8aa-eef309537220\") " pod="openshift-ingress/payload-processing-e2e-api-a-69932e-6d756dcf5d-c8w5p" Oct 30 19:52:32.188702 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:32.188670 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl4v9\" (UniqueName: \"kubernetes.io/projected/3729f9f3-1ad0-48c2-a342-495c57602e6f-kube-api-access-wl4v9\") pod \"payload-pre-processing-e2e-api-a-69932e-774bbd5475-gvztq\" (UID: \"3729f9f3-1ad0-48c2-a342-495c57602e6f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-69932e-774bbd5475-gvztq" Oct 30 19:52:32.188876 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:32.188803 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3729f9f3-1ad0-48c2-a342-495c57602e6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-69932e-774bbd5475-gvztq\" (UID: \"3729f9f3-1ad0-48c2-a342-495c57602e6f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-69932e-774bbd5475-gvztq" Oct 30 19:52:32.290040 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:32.289999 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3729f9f3-1ad0-48c2-a342-495c57602e6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-69932e-774bbd5475-gvztq\" (UID: \"3729f9f3-1ad0-48c2-a342-495c57602e6f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-69932e-774bbd5475-gvztq" Oct 30 19:52:32.290199 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:32.290146 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl4v9\" (UniqueName: \"kubernetes.io/projected/3729f9f3-1ad0-48c2-a342-495c57602e6f-kube-api-access-wl4v9\") pod \"payload-pre-processing-e2e-api-a-69932e-774bbd5475-gvztq\" (UID: \"3729f9f3-1ad0-48c2-a342-495c57602e6f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-69932e-774bbd5475-gvztq" Oct 30 19:52:32.290781 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:32.290759 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3729f9f3-1ad0-48c2-a342-495c57602e6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-69932e-774bbd5475-gvztq\" (UID: \"3729f9f3-1ad0-48c2-a342-495c57602e6f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-69932e-774bbd5475-gvztq" Oct 30 19:52:32.352841 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:32.352810 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl4v9\" (UniqueName: \"kubernetes.io/projected/3729f9f3-1ad0-48c2-a342-495c57602e6f-kube-api-access-wl4v9\") pod \"payload-pre-processing-e2e-api-a-69932e-774bbd5475-gvztq\" (UID: \"3729f9f3-1ad0-48c2-a342-495c57602e6f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-69932e-774bbd5475-gvztq" Oct 30 19:52:32.370699 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:32.370664 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-69932e-6d756dcf5d-c8w5p" Oct 30 19:52:32.410451 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:32.410341 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-69932e-774bbd5475-gvztq" Oct 30 19:52:32.748764 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:32.748728 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-69932e-6d756dcf5d-c8w5p"] Oct 30 19:52:32.749197 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:52:32.749166 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1f916af_267b_4057_a8aa_eef309537220.slice/crio-a15c35ae1ee4956c480d6d19e1cb9afefd853df649af731c73a0cd506c3c4141 WatchSource:0}: Error finding container a15c35ae1ee4956c480d6d19e1cb9afefd853df649af731c73a0cd506c3c4141: Status 404 returned error can't find the container with id a15c35ae1ee4956c480d6d19e1cb9afefd853df649af731c73a0cd506c3c4141 Oct 30 19:52:32.805268 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:32.805242 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-69932e-774bbd5475-gvztq"] Oct 30 19:52:32.805437 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:52:32.805410 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3729f9f3_1ad0_48c2_a342_495c57602e6f.slice/crio-716c81f1b752422fd978b9c5471fe178591de1a6d31b48ade5ec10f67ea9699a WatchSource:0}: Error finding container 716c81f1b752422fd978b9c5471fe178591de1a6d31b48ade5ec10f67ea9699a: Status 404 returned error can't find the container with id 716c81f1b752422fd978b9c5471fe178591de1a6d31b48ade5ec10f67ea9699a Oct 30 19:52:33.081259 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:33.081222 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-69932e-6d756dcf5d-c8w5p" event={"ID":"a1f916af-267b-4057-a8aa-eef309537220","Type":"ContainerStarted","Data":"c5b8373b2607753494f096ac58f85c07505063489ea406d515466e0c1cd8dc09"} Oct 30 19:52:33.081817 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:33.081283 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-69932e-6d756dcf5d-c8w5p" event={"ID":"a1f916af-267b-4057-a8aa-eef309537220","Type":"ContainerStarted","Data":"a15c35ae1ee4956c480d6d19e1cb9afefd853df649af731c73a0cd506c3c4141"} Oct 30 19:52:33.081817 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:33.081449 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-69932e-6d756dcf5d-c8w5p" Oct 30 19:52:33.083068 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:33.083029 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-69932e-774bbd5475-gvztq" event={"ID":"3729f9f3-1ad0-48c2-a342-495c57602e6f","Type":"ContainerStarted","Data":"4bbc5515e9361f584b13516bb02a3a36327102dc83cb9616f2ffddaeaa207acb"} Oct 30 19:52:33.083169 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:33.083076 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-69932e-774bbd5475-gvztq" event={"ID":"3729f9f3-1ad0-48c2-a342-495c57602e6f","Type":"ContainerStarted","Data":"716c81f1b752422fd978b9c5471fe178591de1a6d31b48ade5ec10f67ea9699a"} Oct 30 19:52:33.083169 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:33.083101 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-69932e-774bbd5475-gvztq" Oct 30 19:52:33.201915 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:33.201855 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-69932e-6d756dcf5d-c8w5p" podStartSLOduration=2.20183449 podStartE2EDuration="2.20183449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:33.198744215 +0000 UTC m=+1656.717059672" watchObservedRunningTime="2026-10-30 19:52:33.20183449 +0000 UTC m=+1656.720149947" Oct 30 19:52:33.340446 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:33.332472 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-69932e-774bbd5475-gvztq" podStartSLOduration=2.3324472419999998 podStartE2EDuration="2.332447242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:33.318855186 +0000 UTC m=+1656.837170659" watchObservedRunningTime="2026-10-30 19:52:33.332447242 +0000 UTC m=+1656.850762702" Oct 30 19:52:43.223452 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:52:43.223336 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:52:44.092329 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:44.092294 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-69932e-6d756dcf5d-c8w5p" Oct 30 19:52:44.092537 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:44.092389 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-69932e-774bbd5475-gvztq" Oct 30 19:52:49.260948 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:49.260909 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp"] Oct 30 19:52:49.261463 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:49.261177 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp" podUID="34aa822f-7a67-4e27-9acb-6e8b0b80a5cb" containerName="istio-proxy" containerID="cri-o://1dd48400431929c6a72fb963b6a448ac7c4fb9f534f394f4af7231e53f2a8e8c" gracePeriod=30 Oct 30 19:52:54.527024 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:54.526957 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp" Oct 30 19:52:54.612655 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:54.612614 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb-istio-envoy\") pod \"34aa822f-7a67-4e27-9acb-6e8b0b80a5cb\" (UID: \"34aa822f-7a67-4e27-9acb-6e8b0b80a5cb\") " Oct 30 19:52:54.612860 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:54.612706 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb-istio-data\") pod \"34aa822f-7a67-4e27-9acb-6e8b0b80a5cb\" (UID: \"34aa822f-7a67-4e27-9acb-6e8b0b80a5cb\") " Oct 30 19:52:54.612860 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:54.612732 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb-credential-socket\") pod \"34aa822f-7a67-4e27-9acb-6e8b0b80a5cb\" (UID: \"34aa822f-7a67-4e27-9acb-6e8b0b80a5cb\") " Oct 30 19:52:54.612860 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:54.612762 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb-workload-socket\") pod \"34aa822f-7a67-4e27-9acb-6e8b0b80a5cb\" (UID: \"34aa822f-7a67-4e27-9acb-6e8b0b80a5cb\") " Oct 30 19:52:54.612860 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:54.612793 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb-istio-podinfo\") pod \"34aa822f-7a67-4e27-9acb-6e8b0b80a5cb\" (UID: \"34aa822f-7a67-4e27-9acb-6e8b0b80a5cb\") " Oct 30 19:52:54.613070 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:54.612937 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb-istio-token\" (UniqueName: \"kubernetes.io/projected/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb-istio-token\") pod \"34aa822f-7a67-4e27-9acb-6e8b0b80a5cb\" (UID: \"34aa822f-7a67-4e27-9acb-6e8b0b80a5cb\") " Oct 30 19:52:54.613070 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:54.613047 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb-workload-certs\") pod \"34aa822f-7a67-4e27-9acb-6e8b0b80a5cb\" (UID: \"34aa822f-7a67-4e27-9acb-6e8b0b80a5cb\") " Oct 30 19:52:54.613173 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:54.613072 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb-istio-data" pod "34aa822f-7a67-4e27-9acb-6e8b0b80a5cb" (UID: "34aa822f-7a67-4e27-9acb-6e8b0b80a5cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:52:54.613173 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:54.613080 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb-credential-socket" pod "34aa822f-7a67-4e27-9acb-6e8b0b80a5cb" (UID: "34aa822f-7a67-4e27-9acb-6e8b0b80a5cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:52:54.613173 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:54.613110 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb-workload-socket" pod "34aa822f-7a67-4e27-9acb-6e8b0b80a5cb" (UID: "34aa822f-7a67-4e27-9acb-6e8b0b80a5cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:52:54.613173 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:54.613118 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb-istiod-ca-cert\") pod \"34aa822f-7a67-4e27-9acb-6e8b0b80a5cb\" (UID: \"34aa822f-7a67-4e27-9acb-6e8b0b80a5cb\") " Oct 30 19:52:54.613388 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:54.613182 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb-kube-api-access-gf5vx\" (UniqueName: \"kubernetes.io/projected/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb-kube-api-access-gf5vx\") pod \"34aa822f-7a67-4e27-9acb-6e8b0b80a5cb\" (UID: \"34aa822f-7a67-4e27-9acb-6e8b0b80a5cb\") " Oct 30 19:52:54.613388 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:54.613337 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb-workload-certs" pod "34aa822f-7a67-4e27-9acb-6e8b0b80a5cb" (UID: "34aa822f-7a67-4e27-9acb-6e8b0b80a5cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:52:54.613579 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:54.613553 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb-istiod-ca-cert" pod "34aa822f-7a67-4e27-9acb-6e8b0b80a5cb" (UID: "34aa822f-7a67-4e27-9acb-6e8b0b80a5cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:52:54.613973 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:54.613943 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb-workload-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:52:54.614081 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:54.613976 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb-istiod-ca-cert\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:52:54.614081 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:54.613992 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb-istio-data\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:52:54.614081 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:54.614007 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb-credential-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:52:54.614081 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:54.614022 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb-workload-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:52:54.615072 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:54.615043 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb-istio-envoy" pod "34aa822f-7a67-4e27-9acb-6e8b0b80a5cb" (UID: "34aa822f-7a67-4e27-9acb-6e8b0b80a5cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:52:54.615173 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:54.615088 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb-istio-token" pod "34aa822f-7a67-4e27-9acb-6e8b0b80a5cb" (UID: "34aa822f-7a67-4e27-9acb-6e8b0b80a5cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:52:54.615173 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:54.615106 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb-istio-podinfo" pod "34aa822f-7a67-4e27-9acb-6e8b0b80a5cb" (UID: "34aa822f-7a67-4e27-9acb-6e8b0b80a5cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:52:54.615265 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:54.615253 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb-kube-api-access-gf5vx" pod "34aa822f-7a67-4e27-9acb-6e8b0b80a5cb" (UID: "34aa822f-7a67-4e27-9acb-6e8b0b80a5cb"). InnerVolumeSpecName "kube-api-access-gf5vx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:52:54.719150 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:54.719114 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf5vx\" (UniqueName: \"kubernetes.io/projected/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb-kube-api-access-gf5vx\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:52:54.719150 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:54.719148 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb-istio-envoy\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:52:54.719150 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:54.719158 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb-istio-podinfo\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:52:54.719522 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:54.719167 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb-istio-token\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:52:55.218023 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:55.217983 4292 generic.go:356] "Generic (PLEG): container finished" podID="34aa822f-7a67-4e27-9acb-6e8b0b80a5cb" containerID="1dd48400431929c6a72fb963b6a448ac7c4fb9f534f394f4af7231e53f2a8e8c" exitCode=0 Oct 30 19:52:55.218258 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:55.218066 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp" Oct 30 19:52:55.218258 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:55.218066 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp" event={"ID":"34aa822f-7a67-4e27-9acb-6e8b0b80a5cb","Type":"ContainerDied","Data":"1dd48400431929c6a72fb963b6a448ac7c4fb9f534f394f4af7231e53f2a8e8c"} Oct 30 19:52:55.218258 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:55.218133 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp" event={"ID":"34aa822f-7a67-4e27-9acb-6e8b0b80a5cb","Type":"ContainerDied","Data":"a28a391acc91330fd35267147fe8d82f337a43d59735e119c957b27045e85b7e"} Oct 30 19:52:55.218258 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:55.218167 4292 scope.go:122] "RemoveContainer" containerID="1dd48400431929c6a72fb963b6a448ac7c4fb9f534f394f4af7231e53f2a8e8c" Oct 30 19:52:55.230090 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:55.230074 4292 scope.go:122] "RemoveContainer" containerID="1dd48400431929c6a72fb963b6a448ac7c4fb9f534f394f4af7231e53f2a8e8c" Oct 30 19:52:55.230426 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:52:55.230404 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dd48400431929c6a72fb963b6a448ac7c4fb9f534f394f4af7231e53f2a8e8c\": container with ID starting with 1dd48400431929c6a72fb963b6a448ac7c4fb9f534f394f4af7231e53f2a8e8c not found: ID does not exist" containerID="1dd48400431929c6a72fb963b6a448ac7c4fb9f534f394f4af7231e53f2a8e8c" Oct 30 19:52:55.230527 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:55.230431 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dd48400431929c6a72fb963b6a448ac7c4fb9f534f394f4af7231e53f2a8e8c"} err="failed to get container status \"1dd48400431929c6a72fb963b6a448ac7c4fb9f534f394f4af7231e53f2a8e8c\": rpc error: code = NotFound desc = could not find container \"1dd48400431929c6a72fb963b6a448ac7c4fb9f534f394f4af7231e53f2a8e8c\": container with ID starting with 1dd48400431929c6a72fb963b6a448ac7c4fb9f534f394f4af7231e53f2a8e8c not found: ID does not exist" Oct 30 19:52:55.388669 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:55.388633 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp"] Oct 30 19:52:55.443291 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:55.443247 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-8gptp"] Oct 30 19:52:56.584010 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:56.583976 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34aa822f-7a67-4e27-9acb-6e8b0b80a5cb" path="/var/lib/kubelet/pods/34aa822f-7a67-4e27-9acb-6e8b0b80a5cb/volumes" Oct 30 19:52:57.906981 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:57.906941 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-111f9c64/e2e-lifecycle-model-111f9c64-kserve-5ffc645545-lm7mr"] Oct 30 19:52:57.907538 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:57.907523 4292 cpu_manager.go:414] "containerMap: removing container" podUID="34aa822f-7a67-4e27-9acb-6e8b0b80a5cb" containerName="istio-proxy" Oct 30 19:52:57.907590 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:57.907539 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34aa822f-7a67-4e27-9acb-6e8b0b80a5cb" containerName="istio-proxy" Oct 30 19:52:57.907654 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:57.907645 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="34aa822f-7a67-4e27-9acb-6e8b0b80a5cb" containerName="istio-proxy" Oct 30 19:52:57.912472 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:57.912452 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-111f9c64/e2e-lifecycle-model-111f9c64-kserve-5ffc645545-lm7mr" Oct 30 19:52:57.960071 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:57.960030 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d25c22f7-bb65-4300-9332-872a7be76774-dshm\") pod \"e2e-lifecycle-model-111f9c64-kserve-5ffc645545-lm7mr\" (UID: \"d25c22f7-bb65-4300-9332-872a7be76774\") " pod="ai-tenant-e2e-mt-111f9c64/e2e-lifecycle-model-111f9c64-kserve-5ffc645545-lm7mr" Oct 30 19:52:57.960447 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:57.960407 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d25c22f7-bb65-4300-9332-872a7be76774-model-cache\") pod \"e2e-lifecycle-model-111f9c64-kserve-5ffc645545-lm7mr\" (UID: \"d25c22f7-bb65-4300-9332-872a7be76774\") " pod="ai-tenant-e2e-mt-111f9c64/e2e-lifecycle-model-111f9c64-kserve-5ffc645545-lm7mr" Oct 30 19:52:57.960554 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:57.960477 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d25c22f7-bb65-4300-9332-872a7be76774-home\") pod \"e2e-lifecycle-model-111f9c64-kserve-5ffc645545-lm7mr\" (UID: \"d25c22f7-bb65-4300-9332-872a7be76774\") " pod="ai-tenant-e2e-mt-111f9c64/e2e-lifecycle-model-111f9c64-kserve-5ffc645545-lm7mr" Oct 30 19:52:57.960629 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:57.960598 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d25c22f7-bb65-4300-9332-872a7be76774-tls-certs\") pod \"e2e-lifecycle-model-111f9c64-kserve-5ffc645545-lm7mr\" (UID: \"d25c22f7-bb65-4300-9332-872a7be76774\") " pod="ai-tenant-e2e-mt-111f9c64/e2e-lifecycle-model-111f9c64-kserve-5ffc645545-lm7mr" Oct 30 19:52:57.960681 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:57.960660 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76tzf\" (UniqueName: \"kubernetes.io/projected/d25c22f7-bb65-4300-9332-872a7be76774-kube-api-access-76tzf\") pod \"e2e-lifecycle-model-111f9c64-kserve-5ffc645545-lm7mr\" (UID: \"d25c22f7-bb65-4300-9332-872a7be76774\") " pod="ai-tenant-e2e-mt-111f9c64/e2e-lifecycle-model-111f9c64-kserve-5ffc645545-lm7mr" Oct 30 19:52:57.960719 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:57.960702 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d25c22f7-bb65-4300-9332-872a7be76774-tmp-dir\") pod \"e2e-lifecycle-model-111f9c64-kserve-5ffc645545-lm7mr\" (UID: \"d25c22f7-bb65-4300-9332-872a7be76774\") " pod="ai-tenant-e2e-mt-111f9c64/e2e-lifecycle-model-111f9c64-kserve-5ffc645545-lm7mr" Oct 30 19:52:57.968070 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:57.968031 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-111f9c64/e2e-lifecycle-model-111f9c64-kserve-5ffc645545-lm7mr"] Oct 30 19:52:58.013759 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:58.013720 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-111f9c64\"/\"openshift-service-ca.crt\"" Oct 30 19:52:58.013956 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:58.013783 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-111f9c64\"/\"default-dockercfg-c7d27\"" Oct 30 19:52:58.018059 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:58.018033 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-111f9c64\"/\"kube-root-ca.crt\"" Oct 30 19:52:58.018194 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:58.018058 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-111f9c64\"/\"e2e-lifecycle-model-111f9c64-kserve-self-signed-certs\"" Oct 30 19:52:58.094273 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:58.094227 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d25c22f7-bb65-4300-9332-872a7be76774-dshm\") pod \"e2e-lifecycle-model-111f9c64-kserve-5ffc645545-lm7mr\" (UID: \"d25c22f7-bb65-4300-9332-872a7be76774\") " pod="ai-tenant-e2e-mt-111f9c64/e2e-lifecycle-model-111f9c64-kserve-5ffc645545-lm7mr" Oct 30 19:52:58.100019 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:58.099968 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d25c22f7-bb65-4300-9332-872a7be76774-model-cache\") pod \"e2e-lifecycle-model-111f9c64-kserve-5ffc645545-lm7mr\" (UID: \"d25c22f7-bb65-4300-9332-872a7be76774\") " pod="ai-tenant-e2e-mt-111f9c64/e2e-lifecycle-model-111f9c64-kserve-5ffc645545-lm7mr" Oct 30 19:52:58.101076 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:58.101052 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d25c22f7-bb65-4300-9332-872a7be76774-home\") pod \"e2e-lifecycle-model-111f9c64-kserve-5ffc645545-lm7mr\" (UID: \"d25c22f7-bb65-4300-9332-872a7be76774\") " pod="ai-tenant-e2e-mt-111f9c64/e2e-lifecycle-model-111f9c64-kserve-5ffc645545-lm7mr" Oct 30 19:52:58.101714 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:58.101695 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d25c22f7-bb65-4300-9332-872a7be76774-tls-certs\") pod \"e2e-lifecycle-model-111f9c64-kserve-5ffc645545-lm7mr\" (UID: \"d25c22f7-bb65-4300-9332-872a7be76774\") " pod="ai-tenant-e2e-mt-111f9c64/e2e-lifecycle-model-111f9c64-kserve-5ffc645545-lm7mr" Oct 30 19:52:58.102391 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:58.102371 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76tzf\" (UniqueName: \"kubernetes.io/projected/d25c22f7-bb65-4300-9332-872a7be76774-kube-api-access-76tzf\") pod \"e2e-lifecycle-model-111f9c64-kserve-5ffc645545-lm7mr\" (UID: \"d25c22f7-bb65-4300-9332-872a7be76774\") " pod="ai-tenant-e2e-mt-111f9c64/e2e-lifecycle-model-111f9c64-kserve-5ffc645545-lm7mr" Oct 30 19:52:58.102943 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:58.102922 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d25c22f7-bb65-4300-9332-872a7be76774-tmp-dir\") pod \"e2e-lifecycle-model-111f9c64-kserve-5ffc645545-lm7mr\" (UID: \"d25c22f7-bb65-4300-9332-872a7be76774\") " pod="ai-tenant-e2e-mt-111f9c64/e2e-lifecycle-model-111f9c64-kserve-5ffc645545-lm7mr" Oct 30 19:52:58.118634 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:58.100946 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d25c22f7-bb65-4300-9332-872a7be76774-model-cache\") pod \"e2e-lifecycle-model-111f9c64-kserve-5ffc645545-lm7mr\" (UID: \"d25c22f7-bb65-4300-9332-872a7be76774\") " pod="ai-tenant-e2e-mt-111f9c64/e2e-lifecycle-model-111f9c64-kserve-5ffc645545-lm7mr" Oct 30 19:52:58.118893 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:58.101594 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d25c22f7-bb65-4300-9332-872a7be76774-home\") pod \"e2e-lifecycle-model-111f9c64-kserve-5ffc645545-lm7mr\" (UID: \"d25c22f7-bb65-4300-9332-872a7be76774\") " pod="ai-tenant-e2e-mt-111f9c64/e2e-lifecycle-model-111f9c64-kserve-5ffc645545-lm7mr" Oct 30 19:52:58.119012 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:58.118426 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d25c22f7-bb65-4300-9332-872a7be76774-tmp-dir\") pod \"e2e-lifecycle-model-111f9c64-kserve-5ffc645545-lm7mr\" (UID: \"d25c22f7-bb65-4300-9332-872a7be76774\") " pod="ai-tenant-e2e-mt-111f9c64/e2e-lifecycle-model-111f9c64-kserve-5ffc645545-lm7mr" Oct 30 19:52:58.125917 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:58.125844 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d25c22f7-bb65-4300-9332-872a7be76774-tls-certs\") pod \"e2e-lifecycle-model-111f9c64-kserve-5ffc645545-lm7mr\" (UID: \"d25c22f7-bb65-4300-9332-872a7be76774\") " pod="ai-tenant-e2e-mt-111f9c64/e2e-lifecycle-model-111f9c64-kserve-5ffc645545-lm7mr" Oct 30 19:52:58.126338 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:58.126315 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d25c22f7-bb65-4300-9332-872a7be76774-dshm\") pod \"e2e-lifecycle-model-111f9c64-kserve-5ffc645545-lm7mr\" (UID: \"d25c22f7-bb65-4300-9332-872a7be76774\") " pod="ai-tenant-e2e-mt-111f9c64/e2e-lifecycle-model-111f9c64-kserve-5ffc645545-lm7mr" Oct 30 19:52:58.171289 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:58.171195 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76tzf\" (UniqueName: \"kubernetes.io/projected/d25c22f7-bb65-4300-9332-872a7be76774-kube-api-access-76tzf\") pod \"e2e-lifecycle-model-111f9c64-kserve-5ffc645545-lm7mr\" (UID: \"d25c22f7-bb65-4300-9332-872a7be76774\") " pod="ai-tenant-e2e-mt-111f9c64/e2e-lifecycle-model-111f9c64-kserve-5ffc645545-lm7mr" Oct 30 19:52:58.232089 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:58.232048 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-111f9c64/e2e-lifecycle-model-111f9c64-kserve-5ffc645545-lm7mr" Oct 30 19:52:58.632486 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:52:58.632455 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd25c22f7_bb65_4300_9332_872a7be76774.slice/crio-fecdaa8b03b679c52d15a758e729dc42fff8ee886ba6eaa619de0bdd95ce34e3 WatchSource:0}: Error finding container fecdaa8b03b679c52d15a758e729dc42fff8ee886ba6eaa619de0bdd95ce34e3: Status 404 returned error can't find the container with id fecdaa8b03b679c52d15a758e729dc42fff8ee886ba6eaa619de0bdd95ce34e3 Oct 30 19:52:58.632616 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:58.632536 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-111f9c64/e2e-lifecycle-model-111f9c64-kserve-5ffc645545-lm7mr"] Oct 30 19:52:59.248466 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:59.248427 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-111f9c64/e2e-lifecycle-model-111f9c64-kserve-5ffc645545-lm7mr" event={"ID":"d25c22f7-bb65-4300-9332-872a7be76774","Type":"ContainerStarted","Data":"662cc166ec13b7e039b4da8cf9668432e400448d37dbf370249db0fcf41cd6ba"} Oct 30 19:52:59.248849 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:59.248481 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-111f9c64/e2e-lifecycle-model-111f9c64-kserve-5ffc645545-lm7mr" event={"ID":"d25c22f7-bb65-4300-9332-872a7be76774","Type":"ContainerStarted","Data":"fecdaa8b03b679c52d15a758e729dc42fff8ee886ba6eaa619de0bdd95ce34e3"} Oct 30 19:52:59.370988 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:52:59.370928 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-111f9c64/e2e-lifecycle-model-111f9c64-kserve-5ffc645545-lm7mr" podStartSLOduration=2.370912518 podStartE2EDuration="2.370912518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:59.36850145 +0000 UTC m=+1682.886816907" watchObservedRunningTime="2026-10-30 19:52:59.370912518 +0000 UTC m=+1682.889227975" Oct 30 19:53:00.232978 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:53:00.232941 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-111f9c64/e2e-lifecycle-model-111f9c64-kserve-5ffc645545-lm7mr" Oct 30 19:53:01.052565 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:53:01.052529 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b95df7bc-qddlf"] Oct 30 19:53:01.055657 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:53:01.055633 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b95df7bc-qddlf"] Oct 30 19:53:02.583770 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:53:02.583735 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07c36c0a-58d5-4031-855c-bf07dfd047bc" path="/var/lib/kubelet/pods/07c36c0a-58d5-4031-855c-bf07dfd047bc/volumes" Oct 30 19:53:08.232392 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:53:08.232327 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-111f9c64/e2e-lifecycle-model-111f9c64-kserve-5ffc645545-lm7mr" Oct 30 19:53:08.244907 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:53:08.244879 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-111f9c64/e2e-lifecycle-model-111f9c64-kserve-5ffc645545-lm7mr" Oct 30 19:53:08.322716 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:53:08.322688 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-111f9c64/e2e-lifecycle-model-111f9c64-kserve-5ffc645545-lm7mr" Oct 30 19:53:13.223657 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:53:13.223620 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:53:38.808377 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:53:38.807896 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e67696-66c9b6dfdc-g5mtr"] Oct 30 19:53:38.821722 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:53:38.821694 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e67696-66c9b6dfdc-g5mtr" Oct 30 19:53:38.832674 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:53:38.832645 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e67696-66c9b6dfdc-g5mtr"] Oct 30 19:53:38.843904 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:53:38.843868 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68c94745-a215-46df-bdba-d4aa4fd54f0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e67696-66c9b6dfdc-g5mtr\" (UID: \"68c94745-a215-46df-bdba-d4aa4fd54f0e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e67696-66c9b6dfdc-g5mtr" Oct 30 19:53:38.844064 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:53:38.843941 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nxmb\" (UniqueName: \"kubernetes.io/projected/68c94745-a215-46df-bdba-d4aa4fd54f0e-kube-api-access-4nxmb\") pod \"payload-pre-processing-e2e-api-b-e67696-66c9b6dfdc-g5mtr\" (UID: \"68c94745-a215-46df-bdba-d4aa4fd54f0e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e67696-66c9b6dfdc-g5mtr" Oct 30 19:53:38.879492 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:53:38.879460 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-e67696\"" Oct 30 19:53:38.879643 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:53:38.879496 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-e67696-dockercfg-7j9bn\"" Oct 30 19:53:38.975376 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:53:38.936686 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e67696-8f8b6cbd6-nxb2b"] Oct 30 19:53:38.980690 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:53:38.980661 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e67696-8f8b6cbd6-nxb2b" Oct 30 19:53:38.987377 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:53:38.981248 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68c94745-a215-46df-bdba-d4aa4fd54f0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e67696-66c9b6dfdc-g5mtr\" (UID: \"68c94745-a215-46df-bdba-d4aa4fd54f0e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e67696-66c9b6dfdc-g5mtr" Oct 30 19:53:38.987377 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:53:38.981331 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nxmb\" (UniqueName: \"kubernetes.io/projected/68c94745-a215-46df-bdba-d4aa4fd54f0e-kube-api-access-4nxmb\") pod \"payload-pre-processing-e2e-api-b-e67696-66c9b6dfdc-g5mtr\" (UID: \"68c94745-a215-46df-bdba-d4aa4fd54f0e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e67696-66c9b6dfdc-g5mtr" Oct 30 19:53:38.987377 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:53:38.982846 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68c94745-a215-46df-bdba-d4aa4fd54f0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e67696-66c9b6dfdc-g5mtr\" (UID: \"68c94745-a215-46df-bdba-d4aa4fd54f0e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e67696-66c9b6dfdc-g5mtr" Oct 30 19:53:39.111427 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:53:39.104086 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nxmb\" (UniqueName: \"kubernetes.io/projected/68c94745-a215-46df-bdba-d4aa4fd54f0e-kube-api-access-4nxmb\") pod \"payload-pre-processing-e2e-api-b-e67696-66c9b6dfdc-g5mtr\" (UID: \"68c94745-a215-46df-bdba-d4aa4fd54f0e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e67696-66c9b6dfdc-g5mtr" Oct 30 19:53:39.111427 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:53:39.106244 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e67696-8f8b6cbd6-nxb2b"] Oct 30 19:53:39.158047 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:53:39.158013 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e67696-66c9b6dfdc-g5mtr" Oct 30 19:53:39.191270 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:53:39.191219 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwhlk\" (UniqueName: \"kubernetes.io/projected/c34c4b57-da35-48f4-a66f-775e2ebe751d-kube-api-access-dwhlk\") pod \"payload-processing-e2e-api-b-e67696-8f8b6cbd6-nxb2b\" (UID: \"c34c4b57-da35-48f4-a66f-775e2ebe751d\") " pod="openshift-ingress/payload-processing-e2e-api-b-e67696-8f8b6cbd6-nxb2b" Oct 30 19:53:39.191452 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:53:39.191283 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c34c4b57-da35-48f4-a66f-775e2ebe751d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e67696-8f8b6cbd6-nxb2b\" (UID: \"c34c4b57-da35-48f4-a66f-775e2ebe751d\") " pod="openshift-ingress/payload-processing-e2e-api-b-e67696-8f8b6cbd6-nxb2b" Oct 30 19:53:39.292136 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:53:39.292106 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwhlk\" (UniqueName: \"kubernetes.io/projected/c34c4b57-da35-48f4-a66f-775e2ebe751d-kube-api-access-dwhlk\") pod \"payload-processing-e2e-api-b-e67696-8f8b6cbd6-nxb2b\" (UID: \"c34c4b57-da35-48f4-a66f-775e2ebe751d\") " pod="openshift-ingress/payload-processing-e2e-api-b-e67696-8f8b6cbd6-nxb2b" Oct 30 19:53:39.292328 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:53:39.292167 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c34c4b57-da35-48f4-a66f-775e2ebe751d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e67696-8f8b6cbd6-nxb2b\" (UID: \"c34c4b57-da35-48f4-a66f-775e2ebe751d\") " pod="openshift-ingress/payload-processing-e2e-api-b-e67696-8f8b6cbd6-nxb2b" Oct 30 19:53:39.292956 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:53:39.292927 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c34c4b57-da35-48f4-a66f-775e2ebe751d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e67696-8f8b6cbd6-nxb2b\" (UID: \"c34c4b57-da35-48f4-a66f-775e2ebe751d\") " pod="openshift-ingress/payload-processing-e2e-api-b-e67696-8f8b6cbd6-nxb2b" Oct 30 19:53:39.354757 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:53:39.354725 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwhlk\" (UniqueName: \"kubernetes.io/projected/c34c4b57-da35-48f4-a66f-775e2ebe751d-kube-api-access-dwhlk\") pod \"payload-processing-e2e-api-b-e67696-8f8b6cbd6-nxb2b\" (UID: \"c34c4b57-da35-48f4-a66f-775e2ebe751d\") " pod="openshift-ingress/payload-processing-e2e-api-b-e67696-8f8b6cbd6-nxb2b" Oct 30 19:53:39.394853 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:53:39.394770 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e67696-8f8b6cbd6-nxb2b" Oct 30 19:53:39.559816 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:53:39.559786 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e67696-66c9b6dfdc-g5mtr"] Oct 30 19:53:39.560882 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:53:39.560859 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68c94745_a215_46df_bdba_d4aa4fd54f0e.slice/crio-937132bb6f487b1fed63480a6821c394d036eca2041a4621c10076f94ea4d2d6 WatchSource:0}: Error finding container 937132bb6f487b1fed63480a6821c394d036eca2041a4621c10076f94ea4d2d6: Status 404 returned error can't find the container with id 937132bb6f487b1fed63480a6821c394d036eca2041a4621c10076f94ea4d2d6 Oct 30 19:53:39.635622 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:53:39.635579 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e67696-66c9b6dfdc-g5mtr" event={"ID":"68c94745-a215-46df-bdba-d4aa4fd54f0e","Type":"ContainerStarted","Data":"5dfa845923609f3fc71bb0104b1a72d64b8f127d5d72fb0ded04628b22398577"} Oct 30 19:53:39.635759 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:53:39.635648 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e67696-66c9b6dfdc-g5mtr" event={"ID":"68c94745-a215-46df-bdba-d4aa4fd54f0e","Type":"ContainerStarted","Data":"937132bb6f487b1fed63480a6821c394d036eca2041a4621c10076f94ea4d2d6"} Oct 30 19:53:39.635759 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:53:39.635676 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e67696-66c9b6dfdc-g5mtr" Oct 30 19:53:39.753674 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:53:39.753583 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e67696-66c9b6dfdc-g5mtr" podStartSLOduration=1.7535675579999999 podStartE2EDuration="1.753567558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:39.753200899 +0000 UTC m=+1723.271516356" watchObservedRunningTime="2026-10-30 19:53:39.753567558 +0000 UTC m=+1723.271883015" Oct 30 19:53:39.801094 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:53:39.801067 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e67696-8f8b6cbd6-nxb2b"] Oct 30 19:53:39.802124 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:53:39.802095 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc34c4b57_da35_48f4_a66f_775e2ebe751d.slice/crio-ce966a062d5132bd33ea27421b13d3a9fc705cc1c3d64fa66f8bd4c82ff266bb WatchSource:0}: Error finding container ce966a062d5132bd33ea27421b13d3a9fc705cc1c3d64fa66f8bd4c82ff266bb: Status 404 returned error can't find the container with id ce966a062d5132bd33ea27421b13d3a9fc705cc1c3d64fa66f8bd4c82ff266bb Oct 30 19:53:40.644190 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:53:40.644157 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e67696-8f8b6cbd6-nxb2b" event={"ID":"c34c4b57-da35-48f4-a66f-775e2ebe751d","Type":"ContainerStarted","Data":"2acfe388416e6b6fb3e9ba1e8c0569689a277f26f9f65cd625c480052ec0deea"} Oct 30 19:53:40.644653 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:53:40.644228 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-e67696-8f8b6cbd6-nxb2b" Oct 30 19:53:40.644653 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:53:40.644251 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e67696-8f8b6cbd6-nxb2b" event={"ID":"c34c4b57-da35-48f4-a66f-775e2ebe751d","Type":"ContainerStarted","Data":"ce966a062d5132bd33ea27421b13d3a9fc705cc1c3d64fa66f8bd4c82ff266bb"} Oct 30 19:53:40.763698 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:53:40.763647 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-e67696-8f8b6cbd6-nxb2b" podStartSLOduration=2.7636338030000003 podStartE2EDuration="2.763633803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:40.76289104 +0000 UTC m=+1724.281206539" watchObservedRunningTime="2026-10-30 19:53:40.763633803 +0000 UTC m=+1724.281949259" Oct 30 19:53:43.220775 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:53:43.220727 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:53:50.674378 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:53:50.673058 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e67696-66c9b6dfdc-g5mtr" Oct 30 19:53:51.653955 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:53:51.653928 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-e67696-8f8b6cbd6-nxb2b" Oct 30 19:54:01.793920 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:01.793890 4292 generic.go:356] "Generic (PLEG): container finished" podID="02e3cbb1-bde4-4a15-b543-db2b4b820f99" containerID="28ea1d303ec7b4268abcd0b79525c9c6173ea784f25e5c8364afd72e9e299319" exitCode=0 Oct 30 19:54:01.793920 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:01.793924 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c" event={"ID":"02e3cbb1-bde4-4a15-b543-db2b4b820f99","Type":"ContainerDied","Data":"28ea1d303ec7b4268abcd0b79525c9c6173ea784f25e5c8364afd72e9e299319"} Oct 30 19:54:01.794459 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:01.794345 4292 scope.go:122] "RemoveContainer" containerID="28ea1d303ec7b4268abcd0b79525c9c6173ea784f25e5c8364afd72e9e299319" Oct 30 19:54:01.795962 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:01.795936 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:54:01.796048 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:01.796007 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:54:01.796048 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:01.796040 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:54:02.802249 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:02.802213 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c" event={"ID":"02e3cbb1-bde4-4a15-b543-db2b4b820f99","Type":"ContainerStarted","Data":"0c45484666f573a2a3ef6aac32d0a55a797e6b40a1ebc4eb22af939746ab332b"} Oct 30 19:54:03.083428 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:03.083316 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c" Oct 30 19:54:03.083626 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:03.083600 4292 patch_prober.go:29] interesting pod/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 19:54:03.083704 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:03.083672 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c" podUID="02e3cbb1-bde4-4a15-b543-db2b4b820f99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 19:54:04.083582 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:04.083544 4292 patch_prober.go:29] interesting pod/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 19:54:04.084082 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:04.083609 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c" podUID="02e3cbb1-bde4-4a15-b543-db2b4b820f99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 19:54:05.084029 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:05.083991 4292 patch_prober.go:29] interesting pod/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 19:54:05.084450 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:05.084056 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c" podUID="02e3cbb1-bde4-4a15-b543-db2b4b820f99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 19:54:06.083658 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:06.083621 4292 patch_prober.go:29] interesting pod/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 19:54:06.083841 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:06.083688 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c" podUID="02e3cbb1-bde4-4a15-b543-db2b4b820f99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 19:54:07.084003 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:07.083964 4292 patch_prober.go:29] interesting pod/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 19:54:07.084448 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:07.084036 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c" podUID="02e3cbb1-bde4-4a15-b543-db2b4b820f99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 19:54:08.083822 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:08.083783 4292 patch_prober.go:29] interesting pod/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 19:54:08.084016 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:08.083850 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c" podUID="02e3cbb1-bde4-4a15-b543-db2b4b820f99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 19:54:09.088195 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:09.088162 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c" Oct 30 19:54:09.088726 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:09.088600 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c" Oct 30 19:54:09.089265 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:09.089248 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c" Oct 30 19:54:13.222653 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:54:13.222572 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:54:22.082439 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:22.082391 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4"] Oct 30 19:54:22.083137 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:22.082665 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4" podUID="e5cde15b-cd07-4ce9-8b96-95673db304a8" containerName="istio-proxy" containerID="cri-o://afb8e6622f70196ec501d99e73c066e4495f534a9b03d5ddbc205fcc8cc40110" gracePeriod=30 Oct 30 19:54:26.295598 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:26.295558 4292 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:54:26.296001 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:26.295636 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4" podUID="e5cde15b-cd07-4ce9-8b96-95673db304a8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:54:27.350948 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:27.350922 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4" Oct 30 19:54:27.519340 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:27.519311 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5cde15b-cd07-4ce9-8b96-95673db304a8-istio-token\" (UniqueName: \"kubernetes.io/projected/e5cde15b-cd07-4ce9-8b96-95673db304a8-istio-token\") pod \"e5cde15b-cd07-4ce9-8b96-95673db304a8\" (UID: \"e5cde15b-cd07-4ce9-8b96-95673db304a8\") " Oct 30 19:54:27.519509 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:27.519447 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5cde15b-cd07-4ce9-8b96-95673db304a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5cde15b-cd07-4ce9-8b96-95673db304a8-istio-podinfo\") pod \"e5cde15b-cd07-4ce9-8b96-95673db304a8\" (UID: \"e5cde15b-cd07-4ce9-8b96-95673db304a8\") " Oct 30 19:54:27.519509 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:27.519484 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5cde15b-cd07-4ce9-8b96-95673db304a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5cde15b-cd07-4ce9-8b96-95673db304a8-istio-envoy\") pod \"e5cde15b-cd07-4ce9-8b96-95673db304a8\" (UID: \"e5cde15b-cd07-4ce9-8b96-95673db304a8\") " Oct 30 19:54:27.519587 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:27.519513 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5cde15b-cd07-4ce9-8b96-95673db304a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cde15b-cd07-4ce9-8b96-95673db304a8-workload-socket\") pod \"e5cde15b-cd07-4ce9-8b96-95673db304a8\" (UID: \"e5cde15b-cd07-4ce9-8b96-95673db304a8\") " Oct 30 19:54:27.519667 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:27.519645 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5cde15b-cd07-4ce9-8b96-95673db304a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cde15b-cd07-4ce9-8b96-95673db304a8-credential-socket\") pod \"e5cde15b-cd07-4ce9-8b96-95673db304a8\" (UID: \"e5cde15b-cd07-4ce9-8b96-95673db304a8\") " Oct 30 19:54:27.519761 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:27.519743 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5cde15b-cd07-4ce9-8b96-95673db304a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5cde15b-cd07-4ce9-8b96-95673db304a8-workload-certs\") pod \"e5cde15b-cd07-4ce9-8b96-95673db304a8\" (UID: \"e5cde15b-cd07-4ce9-8b96-95673db304a8\") " Oct 30 19:54:27.519816 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:27.519787 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5cde15b-cd07-4ce9-8b96-95673db304a8-workload-socket" pod "e5cde15b-cd07-4ce9-8b96-95673db304a8" (UID: "e5cde15b-cd07-4ce9-8b96-95673db304a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:54:27.519816 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:27.519791 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5cde15b-cd07-4ce9-8b96-95673db304a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5cde15b-cd07-4ce9-8b96-95673db304a8-istio-data\") pod \"e5cde15b-cd07-4ce9-8b96-95673db304a8\" (UID: \"e5cde15b-cd07-4ce9-8b96-95673db304a8\") " Oct 30 19:54:27.519978 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:27.519926 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5cde15b-cd07-4ce9-8b96-95673db304a8-credential-socket" pod "e5cde15b-cd07-4ce9-8b96-95673db304a8" (UID: "e5cde15b-cd07-4ce9-8b96-95673db304a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:54:27.520063 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:27.519955 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5cde15b-cd07-4ce9-8b96-95673db304a8-istio-data" pod "e5cde15b-cd07-4ce9-8b96-95673db304a8" (UID: "e5cde15b-cd07-4ce9-8b96-95673db304a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:54:27.520063 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:27.519947 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5cde15b-cd07-4ce9-8b96-95673db304a8-kube-api-access-b78qj\" (UniqueName: \"kubernetes.io/projected/e5cde15b-cd07-4ce9-8b96-95673db304a8-kube-api-access-b78qj\") pod \"e5cde15b-cd07-4ce9-8b96-95673db304a8\" (UID: \"e5cde15b-cd07-4ce9-8b96-95673db304a8\") " Oct 30 19:54:27.520063 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:27.520020 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5cde15b-cd07-4ce9-8b96-95673db304a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5cde15b-cd07-4ce9-8b96-95673db304a8-istiod-ca-cert\") pod \"e5cde15b-cd07-4ce9-8b96-95673db304a8\" (UID: \"e5cde15b-cd07-4ce9-8b96-95673db304a8\") " Oct 30 19:54:27.520063 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:27.520017 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5cde15b-cd07-4ce9-8b96-95673db304a8-workload-certs" pod "e5cde15b-cd07-4ce9-8b96-95673db304a8" (UID: "e5cde15b-cd07-4ce9-8b96-95673db304a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:54:27.520337 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:27.520282 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5cde15b-cd07-4ce9-8b96-95673db304a8-istiod-ca-cert" pod "e5cde15b-cd07-4ce9-8b96-95673db304a8" (UID: "e5cde15b-cd07-4ce9-8b96-95673db304a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:54:27.520833 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:27.520807 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5cde15b-cd07-4ce9-8b96-95673db304a8-istiod-ca-cert\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:27.520960 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:27.520837 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cde15b-cd07-4ce9-8b96-95673db304a8-workload-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:27.520960 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:27.520853 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cde15b-cd07-4ce9-8b96-95673db304a8-credential-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:27.520960 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:27.520872 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5cde15b-cd07-4ce9-8b96-95673db304a8-workload-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:27.520960 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:27.520887 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5cde15b-cd07-4ce9-8b96-95673db304a8-istio-data\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:27.521843 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:27.521816 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5cde15b-cd07-4ce9-8b96-95673db304a8-istio-podinfo" pod "e5cde15b-cd07-4ce9-8b96-95673db304a8" (UID: "e5cde15b-cd07-4ce9-8b96-95673db304a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:54:27.521957 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:27.521870 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5cde15b-cd07-4ce9-8b96-95673db304a8-istio-token" pod "e5cde15b-cd07-4ce9-8b96-95673db304a8" (UID: "e5cde15b-cd07-4ce9-8b96-95673db304a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:54:27.521957 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:27.521903 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5cde15b-cd07-4ce9-8b96-95673db304a8-istio-envoy" pod "e5cde15b-cd07-4ce9-8b96-95673db304a8" (UID: "e5cde15b-cd07-4ce9-8b96-95673db304a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:54:27.522099 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:27.522084 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5cde15b-cd07-4ce9-8b96-95673db304a8-kube-api-access-b78qj" pod "e5cde15b-cd07-4ce9-8b96-95673db304a8" (UID: "e5cde15b-cd07-4ce9-8b96-95673db304a8"). InnerVolumeSpecName "kube-api-access-b78qj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:54:27.623101 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:27.623065 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5cde15b-cd07-4ce9-8b96-95673db304a8-istio-podinfo\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:27.623101 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:27.623101 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5cde15b-cd07-4ce9-8b96-95673db304a8-istio-envoy\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:27.623292 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:27.623110 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b78qj\" (UniqueName: \"kubernetes.io/projected/e5cde15b-cd07-4ce9-8b96-95673db304a8-kube-api-access-b78qj\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:27.623292 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:27.623120 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5cde15b-cd07-4ce9-8b96-95673db304a8-istio-token\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:28.015075 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:28.014996 4292 generic.go:356] "Generic (PLEG): container finished" podID="e5cde15b-cd07-4ce9-8b96-95673db304a8" containerID="afb8e6622f70196ec501d99e73c066e4495f534a9b03d5ddbc205fcc8cc40110" exitCode=0 Oct 30 19:54:28.015220 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:28.015092 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4" Oct 30 19:54:28.015220 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:28.015128 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4" event={"ID":"e5cde15b-cd07-4ce9-8b96-95673db304a8","Type":"ContainerDied","Data":"afb8e6622f70196ec501d99e73c066e4495f534a9b03d5ddbc205fcc8cc40110"} Oct 30 19:54:28.015220 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:28.015178 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4" event={"ID":"e5cde15b-cd07-4ce9-8b96-95673db304a8","Type":"ContainerDied","Data":"4b0d7da8ffcd2fce51fbf0c1ecdb05de69026f9b4964b8770ac1df7c60d40992"} Oct 30 19:54:28.015220 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:28.015199 4292 scope.go:122] "RemoveContainer" containerID="afb8e6622f70196ec501d99e73c066e4495f534a9b03d5ddbc205fcc8cc40110" Oct 30 19:54:28.025460 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:28.025399 4292 scope.go:122] "RemoveContainer" containerID="afb8e6622f70196ec501d99e73c066e4495f534a9b03d5ddbc205fcc8cc40110" Oct 30 19:54:28.025669 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:54:28.025652 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afb8e6622f70196ec501d99e73c066e4495f534a9b03d5ddbc205fcc8cc40110\": container with ID starting with afb8e6622f70196ec501d99e73c066e4495f534a9b03d5ddbc205fcc8cc40110 not found: ID does not exist" containerID="afb8e6622f70196ec501d99e73c066e4495f534a9b03d5ddbc205fcc8cc40110" Oct 30 19:54:28.025713 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:28.025676 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afb8e6622f70196ec501d99e73c066e4495f534a9b03d5ddbc205fcc8cc40110"} err="failed to get container status \"afb8e6622f70196ec501d99e73c066e4495f534a9b03d5ddbc205fcc8cc40110\": rpc error: code = NotFound desc = could not find container \"afb8e6622f70196ec501d99e73c066e4495f534a9b03d5ddbc205fcc8cc40110\": container with ID starting with afb8e6622f70196ec501d99e73c066e4495f534a9b03d5ddbc205fcc8cc40110 not found: ID does not exist" Oct 30 19:54:28.185763 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:28.185732 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4"] Oct 30 19:54:28.236429 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:28.236399 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-jmhj4"] Oct 30 19:54:28.584218 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:28.584185 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5cde15b-cd07-4ce9-8b96-95673db304a8" path="/var/lib/kubelet/pods/e5cde15b-cd07-4ce9-8b96-95673db304a8/volumes" Oct 30 19:54:31.830325 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:31.830289 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e67696-6796dcfcf6-zncj4"] Oct 30 19:54:31.831100 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:31.831076 4292 cpu_manager.go:414] "containerMap: removing container" podUID="e5cde15b-cd07-4ce9-8b96-95673db304a8" containerName="istio-proxy" Oct 30 19:54:31.831207 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:31.831101 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5cde15b-cd07-4ce9-8b96-95673db304a8" containerName="istio-proxy" Oct 30 19:54:31.831288 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:31.831275 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5cde15b-cd07-4ce9-8b96-95673db304a8" containerName="istio-proxy" Oct 30 19:54:31.834636 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:31.834614 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e67696-6796dcfcf6-zncj4" Oct 30 19:54:31.890466 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:31.890437 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e67696-6796dcfcf6-zncj4"] Oct 30 19:54:31.892680 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:31.892648 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67pps\" (UniqueName: \"kubernetes.io/projected/b6c13eb3-0acd-4a2e-9171-28042161aa6f-kube-api-access-67pps\") pod \"maas-api-e2e-api-b-e67696-6796dcfcf6-zncj4\" (UID: \"b6c13eb3-0acd-4a2e-9171-28042161aa6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e67696-6796dcfcf6-zncj4" Oct 30 19:54:31.892829 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:31.892786 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6c13eb3-0acd-4a2e-9171-28042161aa6f-metrics-certs\") pod \"maas-api-e2e-api-b-e67696-6796dcfcf6-zncj4\" (UID: \"b6c13eb3-0acd-4a2e-9171-28042161aa6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e67696-6796dcfcf6-zncj4" Oct 30 19:54:31.892980 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:31.892959 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6c13eb3-0acd-4a2e-9171-28042161aa6f-openshift-service-ca\") pod \"maas-api-e2e-api-b-e67696-6796dcfcf6-zncj4\" (UID: \"b6c13eb3-0acd-4a2e-9171-28042161aa6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e67696-6796dcfcf6-zncj4" Oct 30 19:54:31.893063 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:31.893021 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6c13eb3-0acd-4a2e-9171-28042161aa6f-maas-api-tls\") pod \"maas-api-e2e-api-b-e67696-6796dcfcf6-zncj4\" (UID: \"b6c13eb3-0acd-4a2e-9171-28042161aa6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e67696-6796dcfcf6-zncj4" Oct 30 19:54:31.937731 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:31.937702 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-e67696-serving-cert\"" Oct 30 19:54:31.993913 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:31.993882 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6c13eb3-0acd-4a2e-9171-28042161aa6f-openshift-service-ca\") pod \"maas-api-e2e-api-b-e67696-6796dcfcf6-zncj4\" (UID: \"b6c13eb3-0acd-4a2e-9171-28042161aa6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e67696-6796dcfcf6-zncj4" Oct 30 19:54:31.994188 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:31.993937 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6c13eb3-0acd-4a2e-9171-28042161aa6f-maas-api-tls\") pod \"maas-api-e2e-api-b-e67696-6796dcfcf6-zncj4\" (UID: \"b6c13eb3-0acd-4a2e-9171-28042161aa6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e67696-6796dcfcf6-zncj4" Oct 30 19:54:31.994188 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:31.994001 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67pps\" (UniqueName: \"kubernetes.io/projected/b6c13eb3-0acd-4a2e-9171-28042161aa6f-kube-api-access-67pps\") pod \"maas-api-e2e-api-b-e67696-6796dcfcf6-zncj4\" (UID: \"b6c13eb3-0acd-4a2e-9171-28042161aa6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e67696-6796dcfcf6-zncj4" Oct 30 19:54:31.994188 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:31.994065 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6c13eb3-0acd-4a2e-9171-28042161aa6f-metrics-certs\") pod \"maas-api-e2e-api-b-e67696-6796dcfcf6-zncj4\" (UID: \"b6c13eb3-0acd-4a2e-9171-28042161aa6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e67696-6796dcfcf6-zncj4" Oct 30 19:54:31.994188 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:54:31.994105 4292 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-e67696-serving-cert: secret "maas-api-e2e-api-b-e67696-serving-cert" not found Oct 30 19:54:31.994188 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:54:31.994170 4292 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6c13eb3-0acd-4a2e-9171-28042161aa6f-maas-api-tls podName:b6c13eb3-0acd-4a2e-9171-28042161aa6f nodeName:}" failed. No retries permitted until 2026-10-30 19:54:32.494149704 +0000 UTC m=+1776.012465158 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b6c13eb3-0acd-4a2e-9171-28042161aa6f-maas-api-tls") pod "maas-api-e2e-api-b-e67696-6796dcfcf6-zncj4" (UID: "b6c13eb3-0acd-4a2e-9171-28042161aa6f") : secret "maas-api-e2e-api-b-e67696-serving-cert" not found Oct 30 19:54:31.994697 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:31.994671 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6c13eb3-0acd-4a2e-9171-28042161aa6f-openshift-service-ca\") pod \"maas-api-e2e-api-b-e67696-6796dcfcf6-zncj4\" (UID: \"b6c13eb3-0acd-4a2e-9171-28042161aa6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e67696-6796dcfcf6-zncj4" Oct 30 19:54:31.996653 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:31.996625 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6c13eb3-0acd-4a2e-9171-28042161aa6f-metrics-certs\") pod \"maas-api-e2e-api-b-e67696-6796dcfcf6-zncj4\" (UID: \"b6c13eb3-0acd-4a2e-9171-28042161aa6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e67696-6796dcfcf6-zncj4" Oct 30 19:54:32.059670 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:32.059643 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67pps\" (UniqueName: \"kubernetes.io/projected/b6c13eb3-0acd-4a2e-9171-28042161aa6f-kube-api-access-67pps\") pod \"maas-api-e2e-api-b-e67696-6796dcfcf6-zncj4\" (UID: \"b6c13eb3-0acd-4a2e-9171-28042161aa6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e67696-6796dcfcf6-zncj4" Oct 30 19:54:32.503666 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:32.503631 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6c13eb3-0acd-4a2e-9171-28042161aa6f-maas-api-tls\") pod \"maas-api-e2e-api-b-e67696-6796dcfcf6-zncj4\" (UID: \"b6c13eb3-0acd-4a2e-9171-28042161aa6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e67696-6796dcfcf6-zncj4" Oct 30 19:54:32.506062 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:32.506036 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6c13eb3-0acd-4a2e-9171-28042161aa6f-maas-api-tls\") pod \"maas-api-e2e-api-b-e67696-6796dcfcf6-zncj4\" (UID: \"b6c13eb3-0acd-4a2e-9171-28042161aa6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e67696-6796dcfcf6-zncj4" Oct 30 19:54:32.747020 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:32.746988 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e67696-6796dcfcf6-zncj4" Oct 30 19:54:33.125964 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:33.125936 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e67696-6796dcfcf6-zncj4"] Oct 30 19:54:33.127063 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:54:33.127036 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6c13eb3_0acd_4a2e_9171_28042161aa6f.slice/crio-215ce296a0ebd4dea1093b134a54ac44b1711111c6cc4d8e38b1ff1a783685ae WatchSource:0}: Error finding container 215ce296a0ebd4dea1093b134a54ac44b1711111c6cc4d8e38b1ff1a783685ae: Status 404 returned error can't find the container with id 215ce296a0ebd4dea1093b134a54ac44b1711111c6cc4d8e38b1ff1a783685ae Oct 30 19:54:34.051636 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:34.051608 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e67696-6796dcfcf6-zncj4" event={"ID":"b6c13eb3-0acd-4a2e-9171-28042161aa6f","Type":"ContainerStarted","Data":"215ce296a0ebd4dea1093b134a54ac44b1711111c6cc4d8e38b1ff1a783685ae"} Oct 30 19:54:35.059770 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:35.059731 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e67696-6796dcfcf6-zncj4" event={"ID":"b6c13eb3-0acd-4a2e-9171-28042161aa6f","Type":"ContainerStarted","Data":"590b8e96b4ad68e9e46ffccfcc9e537d9fb1ad4d9be5ff138cc28a9785d99b86"} Oct 30 19:54:35.060203 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:35.059801 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e67696-6796dcfcf6-zncj4" Oct 30 19:54:35.181105 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:35.181044 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e67696-6796dcfcf6-zncj4" podStartSLOduration=3.27753225 podStartE2EDuration="4.181031017s" totalImagesPullingTime="903.498767ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:35.17915044 +0000 UTC m=+1778.697465920" watchObservedRunningTime="2026-10-30 19:54:35.181031017 +0000 UTC m=+1778.699346497" Oct 30 19:54:41.071464 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:41.071434 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e67696-6796dcfcf6-zncj4" Oct 30 19:54:43.228579 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:54:43.228527 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:54:50.690569 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:50.690538 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cf283d-b99794875-llrmk"] Oct 30 19:54:51.476533 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:51.476496 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s"] Oct 30 19:54:51.476774 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:51.476750 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s" podUID="dd0da58c-2f5a-4733-bea2-d2599a9931f2" containerName="istio-proxy" containerID="cri-o://444859bc547d4149a131b2775881f880c3e494ec57fefa5ff319364ff223eb8a" gracePeriod=30 Oct 30 19:54:55.691526 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:55.691481 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cf283d-b99794875-llrmk" podUID="730c37c8-3814-42cc-b40e-c557279ad8ca" containerName="payload-pre-processing" containerID="cri-o://3fc5099ae864ecec770d4091c3ead21100c3d89061837b90a3370941e8ce1e7a" gracePeriod=25 Oct 30 19:54:55.711821 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:55.711793 4292 patch_prober.go:29] interesting pod/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:54:55.711943 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:55.711845 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s" podUID="dd0da58c-2f5a-4733-bea2-d2599a9931f2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:54:55.954066 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:55.954008 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cf283d-b99794875-llrmk" Oct 30 19:54:56.068088 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:56.068052 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/730c37c8-3814-42cc-b40e-c557279ad8ca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/730c37c8-3814-42cc-b40e-c557279ad8ca-plugins-config-volume\") pod \"730c37c8-3814-42cc-b40e-c557279ad8ca\" (UID: \"730c37c8-3814-42cc-b40e-c557279ad8ca\") " Oct 30 19:54:56.068590 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:56.068565 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/730c37c8-3814-42cc-b40e-c557279ad8ca-plugins-config-volume" pod "730c37c8-3814-42cc-b40e-c557279ad8ca" (UID: "730c37c8-3814-42cc-b40e-c557279ad8ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:54:56.068723 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:56.068592 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/730c37c8-3814-42cc-b40e-c557279ad8ca-kube-api-access-wk9nr\" (UniqueName: \"kubernetes.io/projected/730c37c8-3814-42cc-b40e-c557279ad8ca-kube-api-access-wk9nr\") pod \"730c37c8-3814-42cc-b40e-c557279ad8ca\" (UID: \"730c37c8-3814-42cc-b40e-c557279ad8ca\") " Oct 30 19:54:56.069141 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:56.069117 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/730c37c8-3814-42cc-b40e-c557279ad8ca-plugins-config-volume\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:56.070900 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:56.070878 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/730c37c8-3814-42cc-b40e-c557279ad8ca-kube-api-access-wk9nr" pod "730c37c8-3814-42cc-b40e-c557279ad8ca" (UID: "730c37c8-3814-42cc-b40e-c557279ad8ca"). InnerVolumeSpecName "kube-api-access-wk9nr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:54:56.170073 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:56.170041 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk9nr\" (UniqueName: \"kubernetes.io/projected/730c37c8-3814-42cc-b40e-c557279ad8ca-kube-api-access-wk9nr\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:56.269901 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:56.269869 4292 generic.go:356] "Generic (PLEG): container finished" podID="730c37c8-3814-42cc-b40e-c557279ad8ca" containerID="3fc5099ae864ecec770d4091c3ead21100c3d89061837b90a3370941e8ce1e7a" exitCode=0 Oct 30 19:54:56.270032 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:56.269920 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cf283d-b99794875-llrmk" Oct 30 19:54:56.270032 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:56.269948 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cf283d-b99794875-llrmk" event={"ID":"730c37c8-3814-42cc-b40e-c557279ad8ca","Type":"ContainerDied","Data":"3fc5099ae864ecec770d4091c3ead21100c3d89061837b90a3370941e8ce1e7a"} Oct 30 19:54:56.270032 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:56.270011 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cf283d-b99794875-llrmk" event={"ID":"730c37c8-3814-42cc-b40e-c557279ad8ca","Type":"ContainerDied","Data":"1f16ff51eef26bfbee85490dc7de47ccd7cef5dfc00f9e36524586e4d8f828f3"} Oct 30 19:54:56.270171 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:56.270035 4292 scope.go:122] "RemoveContainer" containerID="3fc5099ae864ecec770d4091c3ead21100c3d89061837b90a3370941e8ce1e7a" Oct 30 19:54:56.281513 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:56.281450 4292 scope.go:122] "RemoveContainer" containerID="3fc5099ae864ecec770d4091c3ead21100c3d89061837b90a3370941e8ce1e7a" Oct 30 19:54:56.281807 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:54:56.281778 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fc5099ae864ecec770d4091c3ead21100c3d89061837b90a3370941e8ce1e7a\": container with ID starting with 3fc5099ae864ecec770d4091c3ead21100c3d89061837b90a3370941e8ce1e7a not found: ID does not exist" containerID="3fc5099ae864ecec770d4091c3ead21100c3d89061837b90a3370941e8ce1e7a" Oct 30 19:54:56.281924 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:56.281815 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fc5099ae864ecec770d4091c3ead21100c3d89061837b90a3370941e8ce1e7a"} err="failed to get container status \"3fc5099ae864ecec770d4091c3ead21100c3d89061837b90a3370941e8ce1e7a\": rpc error: code = NotFound desc = could not find container \"3fc5099ae864ecec770d4091c3ead21100c3d89061837b90a3370941e8ce1e7a\": container with ID starting with 3fc5099ae864ecec770d4091c3ead21100c3d89061837b90a3370941e8ce1e7a not found: ID does not exist" Oct 30 19:54:56.439268 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:56.439236 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cf283d-b99794875-llrmk"] Oct 30 19:54:56.443342 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:56.443313 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cf283d-b99794875-llrmk"] Oct 30 19:54:56.584339 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:56.584260 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="730c37c8-3814-42cc-b40e-c557279ad8ca" path="/var/lib/kubelet/pods/730c37c8-3814-42cc-b40e-c557279ad8ca/volumes" Oct 30 19:54:56.749459 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:56.749435 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s" Oct 30 19:54:56.839724 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:56.839651 4292 scope.go:122] "RemoveContainer" containerID="82f6512e75490aa5f6e24e5aa087b12a77d3c56a6f2af9d67d7d9bebbd17fb05" Oct 30 19:54:56.848400 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:56.848379 4292 scope.go:122] "RemoveContainer" containerID="444859bc547d4149a131b2775881f880c3e494ec57fefa5ff319364ff223eb8a" Oct 30 19:54:56.878830 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:56.878781 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd0da58c-2f5a-4733-bea2-d2599a9931f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd0da58c-2f5a-4733-bea2-d2599a9931f2-workload-socket\") pod \"dd0da58c-2f5a-4733-bea2-d2599a9931f2\" (UID: \"dd0da58c-2f5a-4733-bea2-d2599a9931f2\") " Oct 30 19:54:56.878945 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:56.878844 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd0da58c-2f5a-4733-bea2-d2599a9931f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd0da58c-2f5a-4733-bea2-d2599a9931f2-workload-certs\") pod \"dd0da58c-2f5a-4733-bea2-d2599a9931f2\" (UID: \"dd0da58c-2f5a-4733-bea2-d2599a9931f2\") " Oct 30 19:54:56.878945 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:56.878895 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd0da58c-2f5a-4733-bea2-d2599a9931f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd0da58c-2f5a-4733-bea2-d2599a9931f2-istio-data\") pod \"dd0da58c-2f5a-4733-bea2-d2599a9931f2\" (UID: \"dd0da58c-2f5a-4733-bea2-d2599a9931f2\") " Oct 30 19:54:56.879080 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:56.879062 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd0da58c-2f5a-4733-bea2-d2599a9931f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd0da58c-2f5a-4733-bea2-d2599a9931f2-istio-podinfo\") pod \"dd0da58c-2f5a-4733-bea2-d2599a9931f2\" (UID: \"dd0da58c-2f5a-4733-bea2-d2599a9931f2\") " Oct 30 19:54:56.879138 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:56.879068 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd0da58c-2f5a-4733-bea2-d2599a9931f2-workload-socket" pod "dd0da58c-2f5a-4733-bea2-d2599a9931f2" (UID: "dd0da58c-2f5a-4733-bea2-d2599a9931f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:54:56.879177 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:56.879155 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd0da58c-2f5a-4733-bea2-d2599a9931f2-workload-certs" pod "dd0da58c-2f5a-4733-bea2-d2599a9931f2" (UID: "dd0da58c-2f5a-4733-bea2-d2599a9931f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:54:56.879230 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:56.879205 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd0da58c-2f5a-4733-bea2-d2599a9931f2-istio-data" pod "dd0da58c-2f5a-4733-bea2-d2599a9931f2" (UID: "dd0da58c-2f5a-4733-bea2-d2599a9931f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:54:56.879461 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:56.879097 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd0da58c-2f5a-4733-bea2-d2599a9931f2-kube-api-access-gzbfl\" (UniqueName: \"kubernetes.io/projected/dd0da58c-2f5a-4733-bea2-d2599a9931f2-kube-api-access-gzbfl\") pod \"dd0da58c-2f5a-4733-bea2-d2599a9931f2\" (UID: \"dd0da58c-2f5a-4733-bea2-d2599a9931f2\") " Oct 30 19:54:56.879600 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:56.879547 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd0da58c-2f5a-4733-bea2-d2599a9931f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd0da58c-2f5a-4733-bea2-d2599a9931f2-istio-envoy\") pod \"dd0da58c-2f5a-4733-bea2-d2599a9931f2\" (UID: \"dd0da58c-2f5a-4733-bea2-d2599a9931f2\") " Oct 30 19:54:56.879600 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:56.879595 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd0da58c-2f5a-4733-bea2-d2599a9931f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd0da58c-2f5a-4733-bea2-d2599a9931f2-credential-socket\") pod \"dd0da58c-2f5a-4733-bea2-d2599a9931f2\" (UID: \"dd0da58c-2f5a-4733-bea2-d2599a9931f2\") " Oct 30 19:54:56.879725 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:56.879657 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd0da58c-2f5a-4733-bea2-d2599a9931f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd0da58c-2f5a-4733-bea2-d2599a9931f2-istiod-ca-cert\") pod \"dd0da58c-2f5a-4733-bea2-d2599a9931f2\" (UID: \"dd0da58c-2f5a-4733-bea2-d2599a9931f2\") " Oct 30 19:54:56.879725 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:56.879683 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd0da58c-2f5a-4733-bea2-d2599a9931f2-istio-token\" (UniqueName: \"kubernetes.io/projected/dd0da58c-2f5a-4733-bea2-d2599a9931f2-istio-token\") pod \"dd0da58c-2f5a-4733-bea2-d2599a9931f2\" (UID: \"dd0da58c-2f5a-4733-bea2-d2599a9931f2\") " Oct 30 19:54:56.880155 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:56.880133 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd0da58c-2f5a-4733-bea2-d2599a9931f2-credential-socket" pod "dd0da58c-2f5a-4733-bea2-d2599a9931f2" (UID: "dd0da58c-2f5a-4733-bea2-d2599a9931f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:54:56.880337 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:56.880268 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd0da58c-2f5a-4733-bea2-d2599a9931f2-istiod-ca-cert" pod "dd0da58c-2f5a-4733-bea2-d2599a9931f2" (UID: "dd0da58c-2f5a-4733-bea2-d2599a9931f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:54:56.880894 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:56.880856 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd0da58c-2f5a-4733-bea2-d2599a9931f2-credential-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:56.880894 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:56.880890 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd0da58c-2f5a-4733-bea2-d2599a9931f2-istiod-ca-cert\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:56.881055 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:56.880904 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd0da58c-2f5a-4733-bea2-d2599a9931f2-workload-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:56.881055 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:56.880917 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd0da58c-2f5a-4733-bea2-d2599a9931f2-workload-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:56.881055 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:56.880930 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd0da58c-2f5a-4733-bea2-d2599a9931f2-istio-data\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:56.881829 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:56.881807 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd0da58c-2f5a-4733-bea2-d2599a9931f2-istio-podinfo" pod "dd0da58c-2f5a-4733-bea2-d2599a9931f2" (UID: "dd0da58c-2f5a-4733-bea2-d2599a9931f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:54:56.884382 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:56.882317 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd0da58c-2f5a-4733-bea2-d2599a9931f2-istio-envoy" pod "dd0da58c-2f5a-4733-bea2-d2599a9931f2" (UID: "dd0da58c-2f5a-4733-bea2-d2599a9931f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:54:56.884382 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:56.882444 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd0da58c-2f5a-4733-bea2-d2599a9931f2-kube-api-access-gzbfl" pod "dd0da58c-2f5a-4733-bea2-d2599a9931f2" (UID: "dd0da58c-2f5a-4733-bea2-d2599a9931f2"). InnerVolumeSpecName "kube-api-access-gzbfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:54:56.884382 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:56.882519 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd0da58c-2f5a-4733-bea2-d2599a9931f2-istio-token" pod "dd0da58c-2f5a-4733-bea2-d2599a9931f2" (UID: "dd0da58c-2f5a-4733-bea2-d2599a9931f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:54:56.983490 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:56.983335 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd0da58c-2f5a-4733-bea2-d2599a9931f2-istio-podinfo\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:56.983490 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:56.983409 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzbfl\" (UniqueName: \"kubernetes.io/projected/dd0da58c-2f5a-4733-bea2-d2599a9931f2-kube-api-access-gzbfl\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:56.983490 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:56.983425 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd0da58c-2f5a-4733-bea2-d2599a9931f2-istio-envoy\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:56.983490 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:56.983438 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd0da58c-2f5a-4733-bea2-d2599a9931f2-istio-token\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:57.276084 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:57.276042 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s" event={"ID":"dd0da58c-2f5a-4733-bea2-d2599a9931f2","Type":"ContainerDied","Data":"444859bc547d4149a131b2775881f880c3e494ec57fefa5ff319364ff223eb8a"} Oct 30 19:54:57.276289 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:57.276095 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s" Oct 30 19:54:57.276289 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:57.276118 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s" event={"ID":"dd0da58c-2f5a-4733-bea2-d2599a9931f2","Type":"ContainerDied","Data":"5ca7d8d7a4215876e88336224028efb69936dd960e27dfea2813428be655e368"} Oct 30 19:54:57.522671 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:57.522637 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s"] Oct 30 19:54:57.526077 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:57.526041 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-p4m6s"] Oct 30 19:54:58.584377 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:54:58.584329 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd0da58c-2f5a-4733-bea2-d2599a9931f2" path="/var/lib/kubelet/pods/dd0da58c-2f5a-4733-bea2-d2599a9931f2/volumes" Oct 30 19:55:13.221531 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:55:13.221495 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:55:21.290404 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:55:21.285773 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-111f9c64-759998544b-n4dkv"] Oct 30 19:55:26.287097 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:55:26.287035 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-111f9c64-759998544b-n4dkv" podUID="d7eb2b8c-8309-4951-82f7-e0f385da96a5" containerName="payload-processing" containerID="cri-o://490b6f311f94a9548131945131f6568de5ef5ce2b96be1b0aec00cb961cfe71c" gracePeriod=25 Oct 30 19:55:26.965237 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:55:26.965195 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-111f9c64-759998544b-n4dkv" podUID="d7eb2b8c-8309-4951-82f7-e0f385da96a5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:55:36.966175 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:55:36.966134 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-111f9c64-759998544b-n4dkv" podUID="d7eb2b8c-8309-4951-82f7-e0f385da96a5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:55:37.341819 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:55:37.341767 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4477e2-995c7ff66-8w9p8"] Oct 30 19:55:37.456101 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:55:37.456060 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4477e2-55fc4bcff9-4x6qk"] Oct 30 19:55:42.342297 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:55:42.342187 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-4477e2-995c7ff66-8w9p8" podUID="56bef2d5-12bf-4ccb-8dea-f031595ddd96" containerName="payload-processing" containerID="cri-o://6fd30c2b0d7cf92edc71af6e275c87a38f7a66ffccb94df682b0abbfeef6496a" gracePeriod=25 Oct 30 19:55:42.457011 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:55:42.456971 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4477e2-55fc4bcff9-4x6qk" podUID="aa4437a1-8344-44c8-91a9-6ae667cbad68" containerName="payload-pre-processing" containerID="cri-o://f8fbc5cb397c17b9c413487a01d8d502bcfda59ecb1cdadf89ec83b0074efb78" gracePeriod=25 Oct 30 19:55:42.617332 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:55:42.617296 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4477e2-995c7ff66-8w9p8" Oct 30 19:55:42.661196 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:55:42.661165 4292 generic.go:356] "Generic (PLEG): container finished" podID="aa4437a1-8344-44c8-91a9-6ae667cbad68" containerID="f8fbc5cb397c17b9c413487a01d8d502bcfda59ecb1cdadf89ec83b0074efb78" exitCode=0 Oct 30 19:55:42.661408 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:55:42.661233 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4477e2-55fc4bcff9-4x6qk" event={"ID":"aa4437a1-8344-44c8-91a9-6ae667cbad68","Type":"ContainerDied","Data":"f8fbc5cb397c17b9c413487a01d8d502bcfda59ecb1cdadf89ec83b0074efb78"} Oct 30 19:55:42.662834 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:55:42.662810 4292 generic.go:356] "Generic (PLEG): container finished" podID="56bef2d5-12bf-4ccb-8dea-f031595ddd96" containerID="6fd30c2b0d7cf92edc71af6e275c87a38f7a66ffccb94df682b0abbfeef6496a" exitCode=0 Oct 30 19:55:42.662965 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:55:42.662862 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4477e2-995c7ff66-8w9p8" Oct 30 19:55:42.662965 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:55:42.662874 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4477e2-995c7ff66-8w9p8" event={"ID":"56bef2d5-12bf-4ccb-8dea-f031595ddd96","Type":"ContainerDied","Data":"6fd30c2b0d7cf92edc71af6e275c87a38f7a66ffccb94df682b0abbfeef6496a"} Oct 30 19:55:42.662965 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:55:42.662917 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4477e2-995c7ff66-8w9p8" event={"ID":"56bef2d5-12bf-4ccb-8dea-f031595ddd96","Type":"ContainerDied","Data":"47863e8f8299ca9d3dd8689873822b46f4e6484131339f6c50ed3da93871a552"} Oct 30 19:55:42.662965 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:55:42.662946 4292 scope.go:122] "RemoveContainer" containerID="6fd30c2b0d7cf92edc71af6e275c87a38f7a66ffccb94df682b0abbfeef6496a" Oct 30 19:55:42.675263 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:55:42.675247 4292 scope.go:122] "RemoveContainer" containerID="6fd30c2b0d7cf92edc71af6e275c87a38f7a66ffccb94df682b0abbfeef6496a" Oct 30 19:55:42.675646 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:55:42.675624 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fd30c2b0d7cf92edc71af6e275c87a38f7a66ffccb94df682b0abbfeef6496a\": container with ID starting with 6fd30c2b0d7cf92edc71af6e275c87a38f7a66ffccb94df682b0abbfeef6496a not found: ID does not exist" containerID="6fd30c2b0d7cf92edc71af6e275c87a38f7a66ffccb94df682b0abbfeef6496a" Oct 30 19:55:42.675755 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:55:42.675652 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fd30c2b0d7cf92edc71af6e275c87a38f7a66ffccb94df682b0abbfeef6496a"} err="failed to get container status \"6fd30c2b0d7cf92edc71af6e275c87a38f7a66ffccb94df682b0abbfeef6496a\": rpc error: code = NotFound desc = could not find container \"6fd30c2b0d7cf92edc71af6e275c87a38f7a66ffccb94df682b0abbfeef6496a\": container with ID starting with 6fd30c2b0d7cf92edc71af6e275c87a38f7a66ffccb94df682b0abbfeef6496a not found: ID does not exist" Oct 30 19:55:42.731246 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:55:42.731014 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4477e2-55fc4bcff9-4x6qk" Oct 30 19:55:42.757381 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:55:42.757328 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56bef2d5-12bf-4ccb-8dea-f031595ddd96-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56bef2d5-12bf-4ccb-8dea-f031595ddd96-plugins-config-volume\") pod \"56bef2d5-12bf-4ccb-8dea-f031595ddd96\" (UID: \"56bef2d5-12bf-4ccb-8dea-f031595ddd96\") " Oct 30 19:55:42.757565 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:55:42.757543 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56bef2d5-12bf-4ccb-8dea-f031595ddd96-kube-api-access-q8q49\" (UniqueName: \"kubernetes.io/projected/56bef2d5-12bf-4ccb-8dea-f031595ddd96-kube-api-access-q8q49\") pod \"56bef2d5-12bf-4ccb-8dea-f031595ddd96\" (UID: \"56bef2d5-12bf-4ccb-8dea-f031595ddd96\") " Oct 30 19:55:42.757729 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:55:42.757706 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56bef2d5-12bf-4ccb-8dea-f031595ddd96-plugins-config-volume" pod "56bef2d5-12bf-4ccb-8dea-f031595ddd96" (UID: "56bef2d5-12bf-4ccb-8dea-f031595ddd96"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:55:42.758064 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:55:42.758042 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56bef2d5-12bf-4ccb-8dea-f031595ddd96-plugins-config-volume\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:42.759659 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:55:42.759633 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56bef2d5-12bf-4ccb-8dea-f031595ddd96-kube-api-access-q8q49" pod "56bef2d5-12bf-4ccb-8dea-f031595ddd96" (UID: "56bef2d5-12bf-4ccb-8dea-f031595ddd96"). InnerVolumeSpecName "kube-api-access-q8q49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:42.859488 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:55:42.859403 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa4437a1-8344-44c8-91a9-6ae667cbad68-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa4437a1-8344-44c8-91a9-6ae667cbad68-plugins-config-volume\") pod \"aa4437a1-8344-44c8-91a9-6ae667cbad68\" (UID: \"aa4437a1-8344-44c8-91a9-6ae667cbad68\") " Oct 30 19:55:42.859658 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:55:42.859545 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa4437a1-8344-44c8-91a9-6ae667cbad68-kube-api-access-v2hrx\" (UniqueName: \"kubernetes.io/projected/aa4437a1-8344-44c8-91a9-6ae667cbad68-kube-api-access-v2hrx\") pod \"aa4437a1-8344-44c8-91a9-6ae667cbad68\" (UID: \"aa4437a1-8344-44c8-91a9-6ae667cbad68\") " Oct 30 19:55:42.859777 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:55:42.859756 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa4437a1-8344-44c8-91a9-6ae667cbad68-plugins-config-volume" pod "aa4437a1-8344-44c8-91a9-6ae667cbad68" (UID: "aa4437a1-8344-44c8-91a9-6ae667cbad68"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:55:42.859988 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:55:42.859970 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa4437a1-8344-44c8-91a9-6ae667cbad68-plugins-config-volume\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:42.860056 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:55:42.859993 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8q49\" (UniqueName: \"kubernetes.io/projected/56bef2d5-12bf-4ccb-8dea-f031595ddd96-kube-api-access-q8q49\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:42.861642 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:55:42.861621 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa4437a1-8344-44c8-91a9-6ae667cbad68-kube-api-access-v2hrx" pod "aa4437a1-8344-44c8-91a9-6ae667cbad68" (UID: "aa4437a1-8344-44c8-91a9-6ae667cbad68"). InnerVolumeSpecName "kube-api-access-v2hrx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:42.961715 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:55:42.961677 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2hrx\" (UniqueName: \"kubernetes.io/projected/aa4437a1-8344-44c8-91a9-6ae667cbad68-kube-api-access-v2hrx\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:43.089823 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:55:43.089788 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4477e2-995c7ff66-8w9p8"] Oct 30 19:55:43.094524 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:55:43.094493 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4477e2-995c7ff66-8w9p8"] Oct 30 19:55:43.221148 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:55:43.221053 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:55:43.671963 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:55:43.671932 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4477e2-55fc4bcff9-4x6qk" event={"ID":"aa4437a1-8344-44c8-91a9-6ae667cbad68","Type":"ContainerDied","Data":"446672aa3760aea0e910f00cf8c6ae7644dd98b2c2f5431c002b6509776f0b45"} Oct 30 19:55:43.672453 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:55:43.671969 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4477e2-55fc4bcff9-4x6qk" Oct 30 19:55:43.672453 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:55:43.671992 4292 scope.go:122] "RemoveContainer" containerID="f8fbc5cb397c17b9c413487a01d8d502bcfda59ecb1cdadf89ec83b0074efb78" Oct 30 19:55:43.840730 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:55:43.840696 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4477e2-55fc4bcff9-4x6qk"] Oct 30 19:55:43.847231 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:55:43.847210 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4477e2-55fc4bcff9-4x6qk"] Oct 30 19:55:44.586600 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:55:44.586557 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56bef2d5-12bf-4ccb-8dea-f031595ddd96" path="/var/lib/kubelet/pods/56bef2d5-12bf-4ccb-8dea-f031595ddd96/volumes" Oct 30 19:55:44.587061 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:55:44.587047 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa4437a1-8344-44c8-91a9-6ae667cbad68" path="/var/lib/kubelet/pods/aa4437a1-8344-44c8-91a9-6ae667cbad68/volumes" Oct 30 19:55:46.545578 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:55:46.545527 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-111f9c64-759998544b-n4dkv" Oct 30 19:55:46.692523 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:55:46.692489 4292 generic.go:356] "Generic (PLEG): container finished" podID="d7eb2b8c-8309-4951-82f7-e0f385da96a5" containerID="490b6f311f94a9548131945131f6568de5ef5ce2b96be1b0aec00cb961cfe71c" exitCode=0 Oct 30 19:55:46.692726 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:55:46.692546 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-111f9c64-759998544b-n4dkv" Oct 30 19:55:46.692726 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:55:46.692572 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-111f9c64-759998544b-n4dkv" event={"ID":"d7eb2b8c-8309-4951-82f7-e0f385da96a5","Type":"ContainerDied","Data":"490b6f311f94a9548131945131f6568de5ef5ce2b96be1b0aec00cb961cfe71c"} Oct 30 19:55:46.692726 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:55:46.692649 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-111f9c64-759998544b-n4dkv" event={"ID":"d7eb2b8c-8309-4951-82f7-e0f385da96a5","Type":"ContainerDied","Data":"629b127329b1a866d687bec8c94513676765c748c7abe5c3dc239cf3955026f0"} Oct 30 19:55:46.692726 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:55:46.692694 4292 scope.go:122] "RemoveContainer" containerID="490b6f311f94a9548131945131f6568de5ef5ce2b96be1b0aec00cb961cfe71c" Oct 30 19:55:46.705412 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:55:46.705388 4292 scope.go:122] "RemoveContainer" containerID="490b6f311f94a9548131945131f6568de5ef5ce2b96be1b0aec00cb961cfe71c" Oct 30 19:55:46.705730 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:55:46.705702 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"490b6f311f94a9548131945131f6568de5ef5ce2b96be1b0aec00cb961cfe71c\": container with ID starting with 490b6f311f94a9548131945131f6568de5ef5ce2b96be1b0aec00cb961cfe71c not found: ID does not exist" containerID="490b6f311f94a9548131945131f6568de5ef5ce2b96be1b0aec00cb961cfe71c" Oct 30 19:55:46.705803 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:55:46.705733 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"490b6f311f94a9548131945131f6568de5ef5ce2b96be1b0aec00cb961cfe71c"} err="failed to get container status \"490b6f311f94a9548131945131f6568de5ef5ce2b96be1b0aec00cb961cfe71c\": rpc error: code = NotFound desc = could not find container \"490b6f311f94a9548131945131f6568de5ef5ce2b96be1b0aec00cb961cfe71c\": container with ID starting with 490b6f311f94a9548131945131f6568de5ef5ce2b96be1b0aec00cb961cfe71c not found: ID does not exist" Oct 30 19:55:46.723996 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:55:46.723977 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7eb2b8c-8309-4951-82f7-e0f385da96a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7eb2b8c-8309-4951-82f7-e0f385da96a5-plugins-config-volume\") pod \"d7eb2b8c-8309-4951-82f7-e0f385da96a5\" (UID: \"d7eb2b8c-8309-4951-82f7-e0f385da96a5\") " Oct 30 19:55:46.724092 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:55:46.724014 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7eb2b8c-8309-4951-82f7-e0f385da96a5-kube-api-access-q6ndx\" (UniqueName: \"kubernetes.io/projected/d7eb2b8c-8309-4951-82f7-e0f385da96a5-kube-api-access-q6ndx\") pod \"d7eb2b8c-8309-4951-82f7-e0f385da96a5\" (UID: \"d7eb2b8c-8309-4951-82f7-e0f385da96a5\") " Oct 30 19:55:46.724302 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:55:46.724282 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7eb2b8c-8309-4951-82f7-e0f385da96a5-plugins-config-volume" pod "d7eb2b8c-8309-4951-82f7-e0f385da96a5" (UID: "d7eb2b8c-8309-4951-82f7-e0f385da96a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:55:46.724461 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:55:46.724446 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7eb2b8c-8309-4951-82f7-e0f385da96a5-plugins-config-volume\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:46.725998 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:55:46.725972 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7eb2b8c-8309-4951-82f7-e0f385da96a5-kube-api-access-q6ndx" pod "d7eb2b8c-8309-4951-82f7-e0f385da96a5" (UID: "d7eb2b8c-8309-4951-82f7-e0f385da96a5"). InnerVolumeSpecName "kube-api-access-q6ndx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:46.825623 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:55:46.825546 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6ndx\" (UniqueName: \"kubernetes.io/projected/d7eb2b8c-8309-4951-82f7-e0f385da96a5-kube-api-access-q6ndx\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:47.120362 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:55:47.120268 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-111f9c64-759998544b-n4dkv"] Oct 30 19:55:47.171137 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:55:47.171104 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-111f9c64-759998544b-n4dkv"] Oct 30 19:55:48.583578 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:55:48.583548 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7eb2b8c-8309-4951-82f7-e0f385da96a5" path="/var/lib/kubelet/pods/d7eb2b8c-8309-4951-82f7-e0f385da96a5/volumes" Oct 30 19:56:00.992002 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:56:00.991881 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-61d11a-7759448859-vzndl"] Oct 30 19:56:05.993142 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:56:05.993082 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-61d11a-7759448859-vzndl" podUID="2bb658ce-897b-46f3-8b91-97c611f6221d" containerName="payload-pre-processing" containerID="cri-o://da530990c6977328117161a7915bc7715d63d57b756d3beb29f63ce890195883" gracePeriod=25 Oct 30 19:56:06.251463 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:56:06.251409 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-61d11a-7759448859-vzndl" Oct 30 19:56:06.381645 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:56:06.381610 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bb658ce-897b-46f3-8b91-97c611f6221d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bb658ce-897b-46f3-8b91-97c611f6221d-plugins-config-volume\") pod \"2bb658ce-897b-46f3-8b91-97c611f6221d\" (UID: \"2bb658ce-897b-46f3-8b91-97c611f6221d\") " Oct 30 19:56:06.381810 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:56:06.381669 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bb658ce-897b-46f3-8b91-97c611f6221d-kube-api-access-hdzqv\" (UniqueName: \"kubernetes.io/projected/2bb658ce-897b-46f3-8b91-97c611f6221d-kube-api-access-hdzqv\") pod \"2bb658ce-897b-46f3-8b91-97c611f6221d\" (UID: \"2bb658ce-897b-46f3-8b91-97c611f6221d\") " Oct 30 19:56:06.382098 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:56:06.382068 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bb658ce-897b-46f3-8b91-97c611f6221d-plugins-config-volume" pod "2bb658ce-897b-46f3-8b91-97c611f6221d" (UID: "2bb658ce-897b-46f3-8b91-97c611f6221d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:56:06.382408 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:56:06.382386 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bb658ce-897b-46f3-8b91-97c611f6221d-plugins-config-volume\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:06.383908 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:56:06.383879 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bb658ce-897b-46f3-8b91-97c611f6221d-kube-api-access-hdzqv" pod "2bb658ce-897b-46f3-8b91-97c611f6221d" (UID: "2bb658ce-897b-46f3-8b91-97c611f6221d"). InnerVolumeSpecName "kube-api-access-hdzqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:56:06.483687 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:56:06.483656 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdzqv\" (UniqueName: \"kubernetes.io/projected/2bb658ce-897b-46f3-8b91-97c611f6221d-kube-api-access-hdzqv\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:06.810339 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:56:06.810305 4292 generic.go:356] "Generic (PLEG): container finished" podID="2bb658ce-897b-46f3-8b91-97c611f6221d" containerID="da530990c6977328117161a7915bc7715d63d57b756d3beb29f63ce890195883" exitCode=0 Oct 30 19:56:06.810556 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:56:06.810375 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-61d11a-7759448859-vzndl" Oct 30 19:56:06.810556 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:56:06.810396 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-61d11a-7759448859-vzndl" event={"ID":"2bb658ce-897b-46f3-8b91-97c611f6221d","Type":"ContainerDied","Data":"da530990c6977328117161a7915bc7715d63d57b756d3beb29f63ce890195883"} Oct 30 19:56:06.810556 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:56:06.810461 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-61d11a-7759448859-vzndl" event={"ID":"2bb658ce-897b-46f3-8b91-97c611f6221d","Type":"ContainerDied","Data":"01b6d6af7f1cb815ddb1a36ccd7976e36b98113ab38fe5aa4d1ab442bea56d6f"} Oct 30 19:56:06.810556 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:56:06.810492 4292 scope.go:122] "RemoveContainer" containerID="da530990c6977328117161a7915bc7715d63d57b756d3beb29f63ce890195883" Oct 30 19:56:06.821891 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:56:06.821867 4292 scope.go:122] "RemoveContainer" containerID="da530990c6977328117161a7915bc7715d63d57b756d3beb29f63ce890195883" Oct 30 19:56:06.822186 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:56:06.822162 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da530990c6977328117161a7915bc7715d63d57b756d3beb29f63ce890195883\": container with ID starting with da530990c6977328117161a7915bc7715d63d57b756d3beb29f63ce890195883 not found: ID does not exist" containerID="da530990c6977328117161a7915bc7715d63d57b756d3beb29f63ce890195883" Oct 30 19:56:06.822258 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:56:06.822192 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da530990c6977328117161a7915bc7715d63d57b756d3beb29f63ce890195883"} err="failed to get container status \"da530990c6977328117161a7915bc7715d63d57b756d3beb29f63ce890195883\": rpc error: code = NotFound desc = could not find container \"da530990c6977328117161a7915bc7715d63d57b756d3beb29f63ce890195883\": container with ID starting with da530990c6977328117161a7915bc7715d63d57b756d3beb29f63ce890195883 not found: ID does not exist" Oct 30 19:56:06.979419 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:56:06.979319 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-61d11a-7759448859-vzndl"] Oct 30 19:56:07.030163 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:56:07.030133 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-61d11a-7759448859-vzndl"] Oct 30 19:56:08.583205 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:56:08.583173 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bb658ce-897b-46f3-8b91-97c611f6221d" path="/var/lib/kubelet/pods/2bb658ce-897b-46f3-8b91-97c611f6221d/volumes" Oct 30 19:56:13.221080 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:56:13.221041 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:56:31.623963 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:56:31.623931 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft"] Oct 30 19:56:31.624624 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:56:31.624139 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft" podUID="caf34ded-76a9-4af9-9b38-bd5437157111" containerName="istio-proxy" containerID="cri-o://7987bd9efcbc925efb40e00456f4d268632ce18de216e258866f2c03687a39e3" gracePeriod=30 Oct 30 19:56:36.877138 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:56:36.877111 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft" Oct 30 19:56:36.930519 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:56:36.930405 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caf34ded-76a9-4af9-9b38-bd5437157111-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf34ded-76a9-4af9-9b38-bd5437157111-workload-socket\") pod \"caf34ded-76a9-4af9-9b38-bd5437157111\" (UID: \"caf34ded-76a9-4af9-9b38-bd5437157111\") " Oct 30 19:56:36.930519 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:56:36.930477 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caf34ded-76a9-4af9-9b38-bd5437157111-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf34ded-76a9-4af9-9b38-bd5437157111-credential-socket\") pod \"caf34ded-76a9-4af9-9b38-bd5437157111\" (UID: \"caf34ded-76a9-4af9-9b38-bd5437157111\") " Oct 30 19:56:36.930750 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:56:36.930539 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/caf34ded-76a9-4af9-9b38-bd5437157111-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caf34ded-76a9-4af9-9b38-bd5437157111-istiod-ca-cert\") pod \"caf34ded-76a9-4af9-9b38-bd5437157111\" (UID: \"caf34ded-76a9-4af9-9b38-bd5437157111\") " Oct 30 19:56:36.930750 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:56:36.930647 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caf34ded-76a9-4af9-9b38-bd5437157111-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caf34ded-76a9-4af9-9b38-bd5437157111-workload-certs\") pod \"caf34ded-76a9-4af9-9b38-bd5437157111\" (UID: \"caf34ded-76a9-4af9-9b38-bd5437157111\") " Oct 30 19:56:36.930750 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:56:36.930670 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caf34ded-76a9-4af9-9b38-bd5437157111-workload-socket" pod "caf34ded-76a9-4af9-9b38-bd5437157111" (UID: "caf34ded-76a9-4af9-9b38-bd5437157111"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:36.930750 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:56:36.930673 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caf34ded-76a9-4af9-9b38-bd5437157111-istio-token\" (UniqueName: \"kubernetes.io/projected/caf34ded-76a9-4af9-9b38-bd5437157111-istio-token\") pod \"caf34ded-76a9-4af9-9b38-bd5437157111\" (UID: \"caf34ded-76a9-4af9-9b38-bd5437157111\") " Oct 30 19:56:36.930961 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:56:36.930758 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caf34ded-76a9-4af9-9b38-bd5437157111-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caf34ded-76a9-4af9-9b38-bd5437157111-istio-data\") pod \"caf34ded-76a9-4af9-9b38-bd5437157111\" (UID: \"caf34ded-76a9-4af9-9b38-bd5437157111\") " Oct 30 19:56:36.930961 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:56:36.930823 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/caf34ded-76a9-4af9-9b38-bd5437157111-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caf34ded-76a9-4af9-9b38-bd5437157111-istio-podinfo\") pod \"caf34ded-76a9-4af9-9b38-bd5437157111\" (UID: \"caf34ded-76a9-4af9-9b38-bd5437157111\") " Oct 30 19:56:36.930961 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:56:36.930795 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caf34ded-76a9-4af9-9b38-bd5437157111-credential-socket" pod "caf34ded-76a9-4af9-9b38-bd5437157111" (UID: "caf34ded-76a9-4af9-9b38-bd5437157111"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:36.930961 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:56:36.930888 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caf34ded-76a9-4af9-9b38-bd5437157111-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caf34ded-76a9-4af9-9b38-bd5437157111-istio-envoy\") pod \"caf34ded-76a9-4af9-9b38-bd5437157111\" (UID: \"caf34ded-76a9-4af9-9b38-bd5437157111\") " Oct 30 19:56:36.930961 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:56:36.930940 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caf34ded-76a9-4af9-9b38-bd5437157111-kube-api-access-cfhtj\" (UniqueName: \"kubernetes.io/projected/caf34ded-76a9-4af9-9b38-bd5437157111-kube-api-access-cfhtj\") pod \"caf34ded-76a9-4af9-9b38-bd5437157111\" (UID: \"caf34ded-76a9-4af9-9b38-bd5437157111\") " Oct 30 19:56:36.931196 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:56:36.930937 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caf34ded-76a9-4af9-9b38-bd5437157111-workload-certs" pod "caf34ded-76a9-4af9-9b38-bd5437157111" (UID: "caf34ded-76a9-4af9-9b38-bd5437157111"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:36.931196 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:56:36.931058 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caf34ded-76a9-4af9-9b38-bd5437157111-istio-data" pod "caf34ded-76a9-4af9-9b38-bd5437157111" (UID: "caf34ded-76a9-4af9-9b38-bd5437157111"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:36.931196 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:56:36.931075 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caf34ded-76a9-4af9-9b38-bd5437157111-istiod-ca-cert" pod "caf34ded-76a9-4af9-9b38-bd5437157111" (UID: "caf34ded-76a9-4af9-9b38-bd5437157111"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:56:36.931573 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:56:36.931548 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caf34ded-76a9-4af9-9b38-bd5437157111-istio-data\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:36.931573 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:56:36.931575 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf34ded-76a9-4af9-9b38-bd5437157111-workload-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:36.931784 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:56:36.931590 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf34ded-76a9-4af9-9b38-bd5437157111-credential-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:36.931784 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:56:36.931601 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caf34ded-76a9-4af9-9b38-bd5437157111-istiod-ca-cert\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:36.931784 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:56:36.931609 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caf34ded-76a9-4af9-9b38-bd5437157111-workload-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:36.932921 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:56:36.932896 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/caf34ded-76a9-4af9-9b38-bd5437157111-istio-podinfo" pod "caf34ded-76a9-4af9-9b38-bd5437157111" (UID: "caf34ded-76a9-4af9-9b38-bd5437157111"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:56:36.933131 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:56:36.933102 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caf34ded-76a9-4af9-9b38-bd5437157111-kube-api-access-cfhtj" pod "caf34ded-76a9-4af9-9b38-bd5437157111" (UID: "caf34ded-76a9-4af9-9b38-bd5437157111"). InnerVolumeSpecName "kube-api-access-cfhtj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:56:36.933218 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:56:36.933150 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caf34ded-76a9-4af9-9b38-bd5437157111-istio-envoy" pod "caf34ded-76a9-4af9-9b38-bd5437157111" (UID: "caf34ded-76a9-4af9-9b38-bd5437157111"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:36.933218 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:56:36.933172 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caf34ded-76a9-4af9-9b38-bd5437157111-istio-token" pod "caf34ded-76a9-4af9-9b38-bd5437157111" (UID: "caf34ded-76a9-4af9-9b38-bd5437157111"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:56:36.974407 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:56:36.974382 4292 generic.go:356] "Generic (PLEG): container finished" podID="caf34ded-76a9-4af9-9b38-bd5437157111" containerID="7987bd9efcbc925efb40e00456f4d268632ce18de216e258866f2c03687a39e3" exitCode=0 Oct 30 19:56:36.974532 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:56:36.974480 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft" Oct 30 19:56:36.974532 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:56:36.974503 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft" event={"ID":"caf34ded-76a9-4af9-9b38-bd5437157111","Type":"ContainerDied","Data":"7987bd9efcbc925efb40e00456f4d268632ce18de216e258866f2c03687a39e3"} Oct 30 19:56:36.974610 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:56:36.974551 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft" event={"ID":"caf34ded-76a9-4af9-9b38-bd5437157111","Type":"ContainerDied","Data":"70a3d7a05c49c99352b52ed185e3ed907c5b42108be37d56c946bee84609309f"} Oct 30 19:56:36.974610 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:56:36.974572 4292 scope.go:122] "RemoveContainer" containerID="7987bd9efcbc925efb40e00456f4d268632ce18de216e258866f2c03687a39e3" Oct 30 19:56:36.984259 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:56:36.984241 4292 scope.go:122] "RemoveContainer" containerID="7987bd9efcbc925efb40e00456f4d268632ce18de216e258866f2c03687a39e3" Oct 30 19:56:36.984650 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:56:36.984632 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7987bd9efcbc925efb40e00456f4d268632ce18de216e258866f2c03687a39e3\": container with ID starting with 7987bd9efcbc925efb40e00456f4d268632ce18de216e258866f2c03687a39e3 not found: ID does not exist" containerID="7987bd9efcbc925efb40e00456f4d268632ce18de216e258866f2c03687a39e3" Oct 30 19:56:36.984735 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:56:36.984660 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7987bd9efcbc925efb40e00456f4d268632ce18de216e258866f2c03687a39e3"} err="failed to get container status \"7987bd9efcbc925efb40e00456f4d268632ce18de216e258866f2c03687a39e3\": rpc error: code = NotFound desc = could not find container \"7987bd9efcbc925efb40e00456f4d268632ce18de216e258866f2c03687a39e3\": container with ID starting with 7987bd9efcbc925efb40e00456f4d268632ce18de216e258866f2c03687a39e3 not found: ID does not exist" Oct 30 19:56:37.033466 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:56:37.033441 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caf34ded-76a9-4af9-9b38-bd5437157111-istio-token\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:37.033466 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:56:37.033463 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caf34ded-76a9-4af9-9b38-bd5437157111-istio-podinfo\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:37.033620 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:56:37.033474 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caf34ded-76a9-4af9-9b38-bd5437157111-istio-envoy\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:37.033620 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:56:37.033483 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfhtj\" (UniqueName: \"kubernetes.io/projected/caf34ded-76a9-4af9-9b38-bd5437157111-kube-api-access-cfhtj\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:37.152291 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:56:37.152257 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft"] Oct 30 19:56:37.156735 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:56:37.156709 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-jv2ft"] Oct 30 19:56:38.592113 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:56:38.592078 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caf34ded-76a9-4af9-9b38-bd5437157111" path="/var/lib/kubelet/pods/caf34ded-76a9-4af9-9b38-bd5437157111/volumes" Oct 30 19:56:43.222429 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:56:43.222388 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:57:09.494393 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:09.494344 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lvnj7"] Oct 30 19:57:09.494952 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:09.494595 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lvnj7" podUID="317c18b6-5b6b-4af5-9ba3-3d5ad719d53b" containerName="istio-proxy" containerID="cri-o://0d0c4377197511e139ad5eb1f0542269cc672c1912d7409d867069d1ff3ff768" gracePeriod=30 Oct 30 19:57:13.223094 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:57:13.223009 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:57:13.621278 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:13.621243 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-lvnj7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:57:13.621485 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:13.621308 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lvnj7" podUID="317c18b6-5b6b-4af5-9ba3-3d5ad719d53b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:57:14.770440 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:14.770413 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lvnj7" Oct 30 19:57:14.862561 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:14.862525 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b-credential-socket\") pod \"317c18b6-5b6b-4af5-9ba3-3d5ad719d53b\" (UID: \"317c18b6-5b6b-4af5-9ba3-3d5ad719d53b\") " Oct 30 19:57:14.862757 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:14.862590 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b-workload-certs\") pod \"317c18b6-5b6b-4af5-9ba3-3d5ad719d53b\" (UID: \"317c18b6-5b6b-4af5-9ba3-3d5ad719d53b\") " Oct 30 19:57:14.862757 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:14.862643 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b-istio-data\") pod \"317c18b6-5b6b-4af5-9ba3-3d5ad719d53b\" (UID: \"317c18b6-5b6b-4af5-9ba3-3d5ad719d53b\") " Oct 30 19:57:14.862757 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:14.862674 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b-workload-socket\") pod \"317c18b6-5b6b-4af5-9ba3-3d5ad719d53b\" (UID: \"317c18b6-5b6b-4af5-9ba3-3d5ad719d53b\") " Oct 30 19:57:14.862757 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:14.862745 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b-kube-api-access-6qjxp\" (UniqueName: \"kubernetes.io/projected/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b-kube-api-access-6qjxp\") pod \"317c18b6-5b6b-4af5-9ba3-3d5ad719d53b\" (UID: \"317c18b6-5b6b-4af5-9ba3-3d5ad719d53b\") " Oct 30 19:57:14.862976 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:14.862853 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b-istio-envoy\") pod \"317c18b6-5b6b-4af5-9ba3-3d5ad719d53b\" (UID: \"317c18b6-5b6b-4af5-9ba3-3d5ad719d53b\") " Oct 30 19:57:14.862976 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:14.862891 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b-istio-token\" (UniqueName: \"kubernetes.io/projected/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b-istio-token\") pod \"317c18b6-5b6b-4af5-9ba3-3d5ad719d53b\" (UID: \"317c18b6-5b6b-4af5-9ba3-3d5ad719d53b\") " Oct 30 19:57:14.862976 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:14.862928 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b-istio-podinfo\") pod \"317c18b6-5b6b-4af5-9ba3-3d5ad719d53b\" (UID: \"317c18b6-5b6b-4af5-9ba3-3d5ad719d53b\") " Oct 30 19:57:14.862976 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:14.862947 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b-istiod-ca-cert\") pod \"317c18b6-5b6b-4af5-9ba3-3d5ad719d53b\" (UID: \"317c18b6-5b6b-4af5-9ba3-3d5ad719d53b\") " Oct 30 19:57:14.863187 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:14.862852 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b-credential-socket" pod "317c18b6-5b6b-4af5-9ba3-3d5ad719d53b" (UID: "317c18b6-5b6b-4af5-9ba3-3d5ad719d53b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:57:14.863187 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:14.862957 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b-workload-certs" pod "317c18b6-5b6b-4af5-9ba3-3d5ad719d53b" (UID: "317c18b6-5b6b-4af5-9ba3-3d5ad719d53b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:57:14.863187 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:14.862977 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b-istio-data" pod "317c18b6-5b6b-4af5-9ba3-3d5ad719d53b" (UID: "317c18b6-5b6b-4af5-9ba3-3d5ad719d53b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:57:14.863187 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:14.863003 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b-workload-socket" pod "317c18b6-5b6b-4af5-9ba3-3d5ad719d53b" (UID: "317c18b6-5b6b-4af5-9ba3-3d5ad719d53b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:57:14.863434 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:14.863399 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b-istiod-ca-cert" pod "317c18b6-5b6b-4af5-9ba3-3d5ad719d53b" (UID: "317c18b6-5b6b-4af5-9ba3-3d5ad719d53b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:57:14.863434 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:14.863421 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b-credential-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:14.863545 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:14.863444 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b-workload-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:14.863545 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:14.863458 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b-istio-data\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:14.863545 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:14.863472 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b-workload-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:14.865153 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:14.865126 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b-istio-podinfo" pod "317c18b6-5b6b-4af5-9ba3-3d5ad719d53b" (UID: "317c18b6-5b6b-4af5-9ba3-3d5ad719d53b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:57:14.865386 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:14.865343 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b-kube-api-access-6qjxp" pod "317c18b6-5b6b-4af5-9ba3-3d5ad719d53b" (UID: "317c18b6-5b6b-4af5-9ba3-3d5ad719d53b"). InnerVolumeSpecName "kube-api-access-6qjxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:57:14.865536 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:14.865458 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b-istio-envoy" pod "317c18b6-5b6b-4af5-9ba3-3d5ad719d53b" (UID: "317c18b6-5b6b-4af5-9ba3-3d5ad719d53b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:57:14.865536 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:14.865500 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b-istio-token" pod "317c18b6-5b6b-4af5-9ba3-3d5ad719d53b" (UID: "317c18b6-5b6b-4af5-9ba3-3d5ad719d53b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:57:14.964269 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:14.964232 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b-istio-envoy\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:14.964269 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:14.964269 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b-istio-token\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:14.964514 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:14.964284 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b-istio-podinfo\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:14.964514 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:14.964296 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b-istiod-ca-cert\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:14.964514 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:14.964309 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qjxp\" (UniqueName: \"kubernetes.io/projected/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b-kube-api-access-6qjxp\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:15.206911 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:15.206820 4292 generic.go:356] "Generic (PLEG): container finished" podID="317c18b6-5b6b-4af5-9ba3-3d5ad719d53b" containerID="0d0c4377197511e139ad5eb1f0542269cc672c1912d7409d867069d1ff3ff768" exitCode=0 Oct 30 19:57:15.206911 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:15.206861 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lvnj7" event={"ID":"317c18b6-5b6b-4af5-9ba3-3d5ad719d53b","Type":"ContainerDied","Data":"0d0c4377197511e139ad5eb1f0542269cc672c1912d7409d867069d1ff3ff768"} Oct 30 19:57:15.206911 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:15.206871 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lvnj7" Oct 30 19:57:15.206911 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:15.206913 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lvnj7" event={"ID":"317c18b6-5b6b-4af5-9ba3-3d5ad719d53b","Type":"ContainerDied","Data":"b3363d40674ccadc3cd8513dcd9d50b8281205aa3adf792267e8ee692e851422"} Oct 30 19:57:15.207241 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:15.206943 4292 scope.go:122] "RemoveContainer" containerID="0d0c4377197511e139ad5eb1f0542269cc672c1912d7409d867069d1ff3ff768" Oct 30 19:57:15.217009 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:15.216987 4292 scope.go:122] "RemoveContainer" containerID="0d0c4377197511e139ad5eb1f0542269cc672c1912d7409d867069d1ff3ff768" Oct 30 19:57:15.217312 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:57:15.217293 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d0c4377197511e139ad5eb1f0542269cc672c1912d7409d867069d1ff3ff768\": container with ID starting with 0d0c4377197511e139ad5eb1f0542269cc672c1912d7409d867069d1ff3ff768 not found: ID does not exist" containerID="0d0c4377197511e139ad5eb1f0542269cc672c1912d7409d867069d1ff3ff768" Oct 30 19:57:15.217435 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:15.217322 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d0c4377197511e139ad5eb1f0542269cc672c1912d7409d867069d1ff3ff768"} err="failed to get container status \"0d0c4377197511e139ad5eb1f0542269cc672c1912d7409d867069d1ff3ff768\": rpc error: code = NotFound desc = could not find container \"0d0c4377197511e139ad5eb1f0542269cc672c1912d7409d867069d1ff3ff768\": container with ID starting with 0d0c4377197511e139ad5eb1f0542269cc672c1912d7409d867069d1ff3ff768 not found: ID does not exist" Oct 30 19:57:15.377457 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:15.377422 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lvnj7"] Oct 30 19:57:15.383433 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:15.383397 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lvnj7"] Oct 30 19:57:16.583233 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:16.583197 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="317c18b6-5b6b-4af5-9ba3-3d5ad719d53b" path="/var/lib/kubelet/pods/317c18b6-5b6b-4af5-9ba3-3d5ad719d53b/volumes" Oct 30 19:57:17.480697 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:17.480662 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4477e2-f97645c84-h7sqc"] Oct 30 19:57:17.480911 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:17.480888 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4477e2-f97645c84-h7sqc" podUID="30db642e-9138-4699-8ffe-7f530f1e956e" containerName="maas-api" containerID="cri-o://b41b10ff578fd659bf128a0d2ebadfb34e3a52533d3836a0f06a8dcdac36d529" gracePeriod=30 Oct 30 19:57:17.832337 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:17.832312 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4477e2-f97645c84-h7sqc" Oct 30 19:57:17.851610 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:17.851581 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30db642e-9138-4699-8ffe-7f530f1e956e-kube-api-access-9rbcr\" (UniqueName: \"kubernetes.io/projected/30db642e-9138-4699-8ffe-7f530f1e956e-kube-api-access-9rbcr\") pod \"30db642e-9138-4699-8ffe-7f530f1e956e\" (UID: \"30db642e-9138-4699-8ffe-7f530f1e956e\") " Oct 30 19:57:17.851783 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:17.851658 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30db642e-9138-4699-8ffe-7f530f1e956e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30db642e-9138-4699-8ffe-7f530f1e956e-maas-api-tls\") pod \"30db642e-9138-4699-8ffe-7f530f1e956e\" (UID: \"30db642e-9138-4699-8ffe-7f530f1e956e\") " Oct 30 19:57:17.851783 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:17.851760 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30db642e-9138-4699-8ffe-7f530f1e956e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/30db642e-9138-4699-8ffe-7f530f1e956e-metrics-certs\") pod \"30db642e-9138-4699-8ffe-7f530f1e956e\" (UID: \"30db642e-9138-4699-8ffe-7f530f1e956e\") " Oct 30 19:57:17.851907 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:17.851793 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30db642e-9138-4699-8ffe-7f530f1e956e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30db642e-9138-4699-8ffe-7f530f1e956e-openshift-service-ca\") pod \"30db642e-9138-4699-8ffe-7f530f1e956e\" (UID: \"30db642e-9138-4699-8ffe-7f530f1e956e\") " Oct 30 19:57:17.852429 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:17.852394 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30db642e-9138-4699-8ffe-7f530f1e956e-openshift-service-ca" pod "30db642e-9138-4699-8ffe-7f530f1e956e" (UID: "30db642e-9138-4699-8ffe-7f530f1e956e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:57:17.853911 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:17.853882 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30db642e-9138-4699-8ffe-7f530f1e956e-maas-api-tls" pod "30db642e-9138-4699-8ffe-7f530f1e956e" (UID: "30db642e-9138-4699-8ffe-7f530f1e956e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:57:17.854305 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:17.854283 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30db642e-9138-4699-8ffe-7f530f1e956e-metrics-certs" pod "30db642e-9138-4699-8ffe-7f530f1e956e" (UID: "30db642e-9138-4699-8ffe-7f530f1e956e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:57:17.854394 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:17.854318 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30db642e-9138-4699-8ffe-7f530f1e956e-kube-api-access-9rbcr" pod "30db642e-9138-4699-8ffe-7f530f1e956e" (UID: "30db642e-9138-4699-8ffe-7f530f1e956e"). InnerVolumeSpecName "kube-api-access-9rbcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:57:17.972003 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:17.968411 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30db642e-9138-4699-8ffe-7f530f1e956e-maas-api-tls\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:17.972003 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:17.968445 4292 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30db642e-9138-4699-8ffe-7f530f1e956e-metrics-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:17.972003 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:17.968458 4292 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30db642e-9138-4699-8ffe-7f530f1e956e-openshift-service-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:17.972003 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:17.968471 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rbcr\" (UniqueName: \"kubernetes.io/projected/30db642e-9138-4699-8ffe-7f530f1e956e-kube-api-access-9rbcr\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:18.255469 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:18.255340 4292 generic.go:356] "Generic (PLEG): container finished" podID="30db642e-9138-4699-8ffe-7f530f1e956e" containerID="b41b10ff578fd659bf128a0d2ebadfb34e3a52533d3836a0f06a8dcdac36d529" exitCode=0 Oct 30 19:57:18.255469 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:18.255405 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4477e2-f97645c84-h7sqc" Oct 30 19:57:18.255469 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:18.255379 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4477e2-f97645c84-h7sqc" event={"ID":"30db642e-9138-4699-8ffe-7f530f1e956e","Type":"ContainerDied","Data":"b41b10ff578fd659bf128a0d2ebadfb34e3a52533d3836a0f06a8dcdac36d529"} Oct 30 19:57:18.255722 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:18.255502 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4477e2-f97645c84-h7sqc" event={"ID":"30db642e-9138-4699-8ffe-7f530f1e956e","Type":"ContainerDied","Data":"8e32d8f982cf54aea279c7233c839f973fd5047d4f13732f23392c099ad13465"} Oct 30 19:57:18.255722 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:18.255546 4292 scope.go:122] "RemoveContainer" containerID="b41b10ff578fd659bf128a0d2ebadfb34e3a52533d3836a0f06a8dcdac36d529" Oct 30 19:57:18.265676 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:18.265658 4292 scope.go:122] "RemoveContainer" containerID="b41b10ff578fd659bf128a0d2ebadfb34e3a52533d3836a0f06a8dcdac36d529" Oct 30 19:57:18.265946 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:57:18.265928 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b41b10ff578fd659bf128a0d2ebadfb34e3a52533d3836a0f06a8dcdac36d529\": container with ID starting with b41b10ff578fd659bf128a0d2ebadfb34e3a52533d3836a0f06a8dcdac36d529 not found: ID does not exist" containerID="b41b10ff578fd659bf128a0d2ebadfb34e3a52533d3836a0f06a8dcdac36d529" Oct 30 19:57:18.265992 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:18.265956 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b41b10ff578fd659bf128a0d2ebadfb34e3a52533d3836a0f06a8dcdac36d529"} err="failed to get container status \"b41b10ff578fd659bf128a0d2ebadfb34e3a52533d3836a0f06a8dcdac36d529\": rpc error: code = NotFound desc = could not find container \"b41b10ff578fd659bf128a0d2ebadfb34e3a52533d3836a0f06a8dcdac36d529\": container with ID starting with b41b10ff578fd659bf128a0d2ebadfb34e3a52533d3836a0f06a8dcdac36d529 not found: ID does not exist" Oct 30 19:57:18.437814 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:18.437773 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4477e2-f97645c84-h7sqc"] Oct 30 19:57:18.488445 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:18.488409 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4477e2-f97645c84-h7sqc"] Oct 30 19:57:18.609688 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:18.609643 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30db642e-9138-4699-8ffe-7f530f1e956e" path="/var/lib/kubelet/pods/30db642e-9138-4699-8ffe-7f530f1e956e/volumes" Oct 30 19:57:18.859878 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:18.859799 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-61d11a-555b777f7d-6zlhz"] Oct 30 19:57:18.860297 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:18.860022 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61d11a-555b777f7d-6zlhz" podUID="fb0e159a-7596-4936-989d-0cc0cdc952af" containerName="maas-api" containerID="cri-o://925c6b8554f72f55e93bb3528b52c7d9fd079e56d878957748a01f480422864a" gracePeriod=30 Oct 30 19:57:19.216489 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:19.216464 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61d11a-555b777f7d-6zlhz" Oct 30 19:57:19.263820 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:19.263785 4292 generic.go:356] "Generic (PLEG): container finished" podID="fb0e159a-7596-4936-989d-0cc0cdc952af" containerID="925c6b8554f72f55e93bb3528b52c7d9fd079e56d878957748a01f480422864a" exitCode=0 Oct 30 19:57:19.263986 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:19.263839 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61d11a-555b777f7d-6zlhz" Oct 30 19:57:19.263986 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:19.263854 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61d11a-555b777f7d-6zlhz" event={"ID":"fb0e159a-7596-4936-989d-0cc0cdc952af","Type":"ContainerDied","Data":"925c6b8554f72f55e93bb3528b52c7d9fd079e56d878957748a01f480422864a"} Oct 30 19:57:19.263986 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:19.263901 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-61d11a-555b777f7d-6zlhz" event={"ID":"fb0e159a-7596-4936-989d-0cc0cdc952af","Type":"ContainerDied","Data":"5c325ddf05e9f2da44b90f43afc75b6b8ada101b6f99f22febcfe888e5ae0a93"} Oct 30 19:57:19.263986 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:19.263927 4292 scope.go:122] "RemoveContainer" containerID="925c6b8554f72f55e93bb3528b52c7d9fd079e56d878957748a01f480422864a" Oct 30 19:57:19.274036 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:19.274017 4292 scope.go:122] "RemoveContainer" containerID="925c6b8554f72f55e93bb3528b52c7d9fd079e56d878957748a01f480422864a" Oct 30 19:57:19.274402 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:57:19.274344 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"925c6b8554f72f55e93bb3528b52c7d9fd079e56d878957748a01f480422864a\": container with ID starting with 925c6b8554f72f55e93bb3528b52c7d9fd079e56d878957748a01f480422864a not found: ID does not exist" containerID="925c6b8554f72f55e93bb3528b52c7d9fd079e56d878957748a01f480422864a" Oct 30 19:57:19.274485 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:19.274415 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"925c6b8554f72f55e93bb3528b52c7d9fd079e56d878957748a01f480422864a"} err="failed to get container status \"925c6b8554f72f55e93bb3528b52c7d9fd079e56d878957748a01f480422864a\": rpc error: code = NotFound desc = could not find container \"925c6b8554f72f55e93bb3528b52c7d9fd079e56d878957748a01f480422864a\": container with ID starting with 925c6b8554f72f55e93bb3528b52c7d9fd079e56d878957748a01f480422864a not found: ID does not exist" Oct 30 19:57:19.337654 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:19.329812 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb0e159a-7596-4936-989d-0cc0cdc952af-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb0e159a-7596-4936-989d-0cc0cdc952af-openshift-service-ca\") pod \"fb0e159a-7596-4936-989d-0cc0cdc952af\" (UID: \"fb0e159a-7596-4936-989d-0cc0cdc952af\") " Oct 30 19:57:19.337654 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:19.329964 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb0e159a-7596-4936-989d-0cc0cdc952af-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb0e159a-7596-4936-989d-0cc0cdc952af-metrics-certs\") pod \"fb0e159a-7596-4936-989d-0cc0cdc952af\" (UID: \"fb0e159a-7596-4936-989d-0cc0cdc952af\") " Oct 30 19:57:19.337654 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:19.330028 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb0e159a-7596-4936-989d-0cc0cdc952af-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb0e159a-7596-4936-989d-0cc0cdc952af-maas-api-tls\") pod \"fb0e159a-7596-4936-989d-0cc0cdc952af\" (UID: \"fb0e159a-7596-4936-989d-0cc0cdc952af\") " Oct 30 19:57:19.337654 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:19.330072 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb0e159a-7596-4936-989d-0cc0cdc952af-kube-api-access-ft2jf\" (UniqueName: \"kubernetes.io/projected/fb0e159a-7596-4936-989d-0cc0cdc952af-kube-api-access-ft2jf\") pod \"fb0e159a-7596-4936-989d-0cc0cdc952af\" (UID: \"fb0e159a-7596-4936-989d-0cc0cdc952af\") " Oct 30 19:57:19.338294 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:19.338191 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb0e159a-7596-4936-989d-0cc0cdc952af-kube-api-access-ft2jf" pod "fb0e159a-7596-4936-989d-0cc0cdc952af" (UID: "fb0e159a-7596-4936-989d-0cc0cdc952af"). InnerVolumeSpecName "kube-api-access-ft2jf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:57:19.356401 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:19.355586 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb0e159a-7596-4936-989d-0cc0cdc952af-openshift-service-ca" pod "fb0e159a-7596-4936-989d-0cc0cdc952af" (UID: "fb0e159a-7596-4936-989d-0cc0cdc952af"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:57:19.382372 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:19.375512 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb0e159a-7596-4936-989d-0cc0cdc952af-metrics-certs" pod "fb0e159a-7596-4936-989d-0cc0cdc952af" (UID: "fb0e159a-7596-4936-989d-0cc0cdc952af"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:57:19.382372 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:19.381767 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb0e159a-7596-4936-989d-0cc0cdc952af-maas-api-tls" pod "fb0e159a-7596-4936-989d-0cc0cdc952af" (UID: "fb0e159a-7596-4936-989d-0cc0cdc952af"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:57:19.432078 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:19.432007 4292 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb0e159a-7596-4936-989d-0cc0cdc952af-openshift-service-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:19.432078 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:19.432035 4292 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb0e159a-7596-4936-989d-0cc0cdc952af-metrics-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:19.432078 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:19.432045 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb0e159a-7596-4936-989d-0cc0cdc952af-maas-api-tls\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:19.432078 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:19.432053 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ft2jf\" (UniqueName: \"kubernetes.io/projected/fb0e159a-7596-4936-989d-0cc0cdc952af-kube-api-access-ft2jf\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:19.689781 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:19.689704 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-61d11a-555b777f7d-6zlhz"] Oct 30 19:57:19.692167 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:19.692141 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-61d11a-555b777f7d-6zlhz"] Oct 30 19:57:20.584543 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:20.584509 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb0e159a-7596-4936-989d-0cc0cdc952af" path="/var/lib/kubelet/pods/fb0e159a-7596-4936-989d-0cc0cdc952af/volumes" Oct 30 19:57:22.982941 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:22.982904 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-69932e-6d756dcf5d-c8w5p"] Oct 30 19:57:23.090866 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:23.090830 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-69932e-774bbd5475-gvztq"] Oct 30 19:57:27.984073 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:27.984028 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-69932e-6d756dcf5d-c8w5p" podUID="a1f916af-267b-4057-a8aa-eef309537220" containerName="payload-processing" containerID="cri-o://c5b8373b2607753494f096ac58f85c07505063489ea406d515466e0c1cd8dc09" gracePeriod=25 Oct 30 19:57:28.091638 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:28.091590 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-69932e-774bbd5475-gvztq" podUID="3729f9f3-1ad0-48c2-a342-495c57602e6f" containerName="payload-pre-processing" containerID="cri-o://4bbc5515e9361f584b13516bb02a3a36327102dc83cb9616f2ffddaeaa207acb" gracePeriod=25 Oct 30 19:57:28.252275 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:28.252206 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-69932e-6d756dcf5d-c8w5p" Oct 30 19:57:28.322456 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:28.322412 4292 generic.go:356] "Generic (PLEG): container finished" podID="a1f916af-267b-4057-a8aa-eef309537220" containerID="c5b8373b2607753494f096ac58f85c07505063489ea406d515466e0c1cd8dc09" exitCode=0 Oct 30 19:57:28.322621 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:28.322480 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-69932e-6d756dcf5d-c8w5p" Oct 30 19:57:28.322621 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:28.322497 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-69932e-6d756dcf5d-c8w5p" event={"ID":"a1f916af-267b-4057-a8aa-eef309537220","Type":"ContainerDied","Data":"c5b8373b2607753494f096ac58f85c07505063489ea406d515466e0c1cd8dc09"} Oct 30 19:57:28.322621 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:28.322551 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-69932e-6d756dcf5d-c8w5p" event={"ID":"a1f916af-267b-4057-a8aa-eef309537220","Type":"ContainerDied","Data":"a15c35ae1ee4956c480d6d19e1cb9afefd853df649af731c73a0cd506c3c4141"} Oct 30 19:57:28.322621 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:28.322584 4292 scope.go:122] "RemoveContainer" containerID="c5b8373b2607753494f096ac58f85c07505063489ea406d515466e0c1cd8dc09" Oct 30 19:57:28.325710 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:28.325684 4292 generic.go:356] "Generic (PLEG): container finished" podID="3729f9f3-1ad0-48c2-a342-495c57602e6f" containerID="4bbc5515e9361f584b13516bb02a3a36327102dc83cb9616f2ffddaeaa207acb" exitCode=0 Oct 30 19:57:28.325844 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:28.325754 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-69932e-774bbd5475-gvztq" event={"ID":"3729f9f3-1ad0-48c2-a342-495c57602e6f","Type":"ContainerDied","Data":"4bbc5515e9361f584b13516bb02a3a36327102dc83cb9616f2ffddaeaa207acb"} Oct 30 19:57:28.333159 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:28.333134 4292 scope.go:122] "RemoveContainer" containerID="c5b8373b2607753494f096ac58f85c07505063489ea406d515466e0c1cd8dc09" Oct 30 19:57:28.333495 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:57:28.333468 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5b8373b2607753494f096ac58f85c07505063489ea406d515466e0c1cd8dc09\": container with ID starting with c5b8373b2607753494f096ac58f85c07505063489ea406d515466e0c1cd8dc09 not found: ID does not exist" containerID="c5b8373b2607753494f096ac58f85c07505063489ea406d515466e0c1cd8dc09" Oct 30 19:57:28.333581 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:28.333506 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5b8373b2607753494f096ac58f85c07505063489ea406d515466e0c1cd8dc09"} err="failed to get container status \"c5b8373b2607753494f096ac58f85c07505063489ea406d515466e0c1cd8dc09\": rpc error: code = NotFound desc = could not find container \"c5b8373b2607753494f096ac58f85c07505063489ea406d515466e0c1cd8dc09\": container with ID starting with c5b8373b2607753494f096ac58f85c07505063489ea406d515466e0c1cd8dc09 not found: ID does not exist" Oct 30 19:57:28.359431 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:28.359409 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-69932e-774bbd5475-gvztq" Oct 30 19:57:28.395862 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:28.395823 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1f916af-267b-4057-a8aa-eef309537220-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1f916af-267b-4057-a8aa-eef309537220-plugins-config-volume\") pod \"a1f916af-267b-4057-a8aa-eef309537220\" (UID: \"a1f916af-267b-4057-a8aa-eef309537220\") " Oct 30 19:57:28.396079 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:28.396054 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1f916af-267b-4057-a8aa-eef309537220-kube-api-access-nb955\" (UniqueName: \"kubernetes.io/projected/a1f916af-267b-4057-a8aa-eef309537220-kube-api-access-nb955\") pod \"a1f916af-267b-4057-a8aa-eef309537220\" (UID: \"a1f916af-267b-4057-a8aa-eef309537220\") " Oct 30 19:57:28.396293 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:28.396261 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1f916af-267b-4057-a8aa-eef309537220-plugins-config-volume" pod "a1f916af-267b-4057-a8aa-eef309537220" (UID: "a1f916af-267b-4057-a8aa-eef309537220"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:57:28.396718 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:28.396696 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1f916af-267b-4057-a8aa-eef309537220-plugins-config-volume\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:28.398685 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:28.398661 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1f916af-267b-4057-a8aa-eef309537220-kube-api-access-nb955" pod "a1f916af-267b-4057-a8aa-eef309537220" (UID: "a1f916af-267b-4057-a8aa-eef309537220"). InnerVolumeSpecName "kube-api-access-nb955". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:57:28.497569 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:28.497534 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3729f9f3-1ad0-48c2-a342-495c57602e6f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3729f9f3-1ad0-48c2-a342-495c57602e6f-plugins-config-volume\") pod \"3729f9f3-1ad0-48c2-a342-495c57602e6f\" (UID: \"3729f9f3-1ad0-48c2-a342-495c57602e6f\") " Oct 30 19:57:28.497726 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:28.497630 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3729f9f3-1ad0-48c2-a342-495c57602e6f-kube-api-access-wl4v9\" (UniqueName: \"kubernetes.io/projected/3729f9f3-1ad0-48c2-a342-495c57602e6f-kube-api-access-wl4v9\") pod \"3729f9f3-1ad0-48c2-a342-495c57602e6f\" (UID: \"3729f9f3-1ad0-48c2-a342-495c57602e6f\") " Oct 30 19:57:28.497933 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:28.497911 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3729f9f3-1ad0-48c2-a342-495c57602e6f-plugins-config-volume" pod "3729f9f3-1ad0-48c2-a342-495c57602e6f" (UID: "3729f9f3-1ad0-48c2-a342-495c57602e6f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:57:28.498059 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:28.498034 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb955\" (UniqueName: \"kubernetes.io/projected/a1f916af-267b-4057-a8aa-eef309537220-kube-api-access-nb955\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:28.498059 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:28.498055 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3729f9f3-1ad0-48c2-a342-495c57602e6f-plugins-config-volume\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:28.499674 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:28.499652 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3729f9f3-1ad0-48c2-a342-495c57602e6f-kube-api-access-wl4v9" pod "3729f9f3-1ad0-48c2-a342-495c57602e6f" (UID: "3729f9f3-1ad0-48c2-a342-495c57602e6f"). InnerVolumeSpecName "kube-api-access-wl4v9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:57:28.599077 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:28.599050 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl4v9\" (UniqueName: \"kubernetes.io/projected/3729f9f3-1ad0-48c2-a342-495c57602e6f-kube-api-access-wl4v9\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:28.756034 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:28.755990 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-69932e-6d756dcf5d-c8w5p"] Oct 30 19:57:28.759343 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:28.759308 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-69932e-6d756dcf5d-c8w5p"] Oct 30 19:57:29.333111 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:29.333085 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-69932e-774bbd5475-gvztq" Oct 30 19:57:29.333636 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:29.333119 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-69932e-774bbd5475-gvztq" event={"ID":"3729f9f3-1ad0-48c2-a342-495c57602e6f","Type":"ContainerDied","Data":"716c81f1b752422fd978b9c5471fe178591de1a6d31b48ade5ec10f67ea9699a"} Oct 30 19:57:29.333636 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:29.333177 4292 scope.go:122] "RemoveContainer" containerID="4bbc5515e9361f584b13516bb02a3a36327102dc83cb9616f2ffddaeaa207acb" Oct 30 19:57:29.504573 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:29.504541 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-69932e-774bbd5475-gvztq"] Oct 30 19:57:29.509467 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:29.509440 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-69932e-774bbd5475-gvztq"] Oct 30 19:57:30.582052 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:30.582019 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3729f9f3-1ad0-48c2-a342-495c57602e6f" path="/var/lib/kubelet/pods/3729f9f3-1ad0-48c2-a342-495c57602e6f/volumes" Oct 30 19:57:30.582484 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:30.582469 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1f916af-267b-4057-a8aa-eef309537220" path="/var/lib/kubelet/pods/a1f916af-267b-4057-a8aa-eef309537220/volumes" Oct 30 19:57:41.378634 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:41.378598 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e67696-8f8b6cbd6-nxb2b"] Oct 30 19:57:41.494894 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:41.494854 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e67696-66c9b6dfdc-g5mtr"] Oct 30 19:57:43.263761 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:57:43.263722 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:57:46.379277 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:46.379234 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-e67696-8f8b6cbd6-nxb2b" podUID="c34c4b57-da35-48f4-a66f-775e2ebe751d" containerName="payload-processing" containerID="cri-o://2acfe388416e6b6fb3e9ba1e8c0569689a277f26f9f65cd625c480052ec0deea" gracePeriod=25 Oct 30 19:57:46.495812 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:46.495761 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e67696-66c9b6dfdc-g5mtr" podUID="68c94745-a215-46df-bdba-d4aa4fd54f0e" containerName="payload-pre-processing" containerID="cri-o://5dfa845923609f3fc71bb0104b1a72d64b8f127d5d72fb0ded04628b22398577" gracePeriod=25 Oct 30 19:57:46.642813 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:46.642742 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e67696-8f8b6cbd6-nxb2b" Oct 30 19:57:46.753916 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:46.753892 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e67696-66c9b6dfdc-g5mtr" Oct 30 19:57:46.772397 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:46.772366 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c34c4b57-da35-48f4-a66f-775e2ebe751d-kube-api-access-dwhlk\" (UniqueName: \"kubernetes.io/projected/c34c4b57-da35-48f4-a66f-775e2ebe751d-kube-api-access-dwhlk\") pod \"c34c4b57-da35-48f4-a66f-775e2ebe751d\" (UID: \"c34c4b57-da35-48f4-a66f-775e2ebe751d\") " Oct 30 19:57:46.772561 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:46.772463 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c34c4b57-da35-48f4-a66f-775e2ebe751d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c34c4b57-da35-48f4-a66f-775e2ebe751d-plugins-config-volume\") pod \"c34c4b57-da35-48f4-a66f-775e2ebe751d\" (UID: \"c34c4b57-da35-48f4-a66f-775e2ebe751d\") " Oct 30 19:57:46.772894 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:46.772868 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c34c4b57-da35-48f4-a66f-775e2ebe751d-plugins-config-volume" pod "c34c4b57-da35-48f4-a66f-775e2ebe751d" (UID: "c34c4b57-da35-48f4-a66f-775e2ebe751d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:57:46.774343 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:46.774323 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c34c4b57-da35-48f4-a66f-775e2ebe751d-kube-api-access-dwhlk" pod "c34c4b57-da35-48f4-a66f-775e2ebe751d" (UID: "c34c4b57-da35-48f4-a66f-775e2ebe751d"). InnerVolumeSpecName "kube-api-access-dwhlk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:57:46.873651 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:46.873621 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68c94745-a215-46df-bdba-d4aa4fd54f0e-kube-api-access-4nxmb\" (UniqueName: \"kubernetes.io/projected/68c94745-a215-46df-bdba-d4aa4fd54f0e-kube-api-access-4nxmb\") pod \"68c94745-a215-46df-bdba-d4aa4fd54f0e\" (UID: \"68c94745-a215-46df-bdba-d4aa4fd54f0e\") " Oct 30 19:57:46.873832 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:46.873741 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68c94745-a215-46df-bdba-d4aa4fd54f0e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68c94745-a215-46df-bdba-d4aa4fd54f0e-plugins-config-volume\") pod \"68c94745-a215-46df-bdba-d4aa4fd54f0e\" (UID: \"68c94745-a215-46df-bdba-d4aa4fd54f0e\") " Oct 30 19:57:46.874169 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:46.874142 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68c94745-a215-46df-bdba-d4aa4fd54f0e-plugins-config-volume" pod "68c94745-a215-46df-bdba-d4aa4fd54f0e" (UID: "68c94745-a215-46df-bdba-d4aa4fd54f0e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:57:46.874292 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:46.874218 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwhlk\" (UniqueName: \"kubernetes.io/projected/c34c4b57-da35-48f4-a66f-775e2ebe751d-kube-api-access-dwhlk\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:46.874292 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:46.874244 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c34c4b57-da35-48f4-a66f-775e2ebe751d-plugins-config-volume\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:46.875748 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:46.875721 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68c94745-a215-46df-bdba-d4aa4fd54f0e-kube-api-access-4nxmb" pod "68c94745-a215-46df-bdba-d4aa4fd54f0e" (UID: "68c94745-a215-46df-bdba-d4aa4fd54f0e"). InnerVolumeSpecName "kube-api-access-4nxmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:57:46.975085 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:46.975006 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68c94745-a215-46df-bdba-d4aa4fd54f0e-plugins-config-volume\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:46.975085 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:46.975035 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nxmb\" (UniqueName: \"kubernetes.io/projected/68c94745-a215-46df-bdba-d4aa4fd54f0e-kube-api-access-4nxmb\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:47.439026 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:47.438993 4292 generic.go:356] "Generic (PLEG): container finished" podID="c34c4b57-da35-48f4-a66f-775e2ebe751d" containerID="2acfe388416e6b6fb3e9ba1e8c0569689a277f26f9f65cd625c480052ec0deea" exitCode=0 Oct 30 19:57:47.439494 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:47.439038 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e67696-8f8b6cbd6-nxb2b" Oct 30 19:57:47.439494 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:47.439080 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e67696-8f8b6cbd6-nxb2b" event={"ID":"c34c4b57-da35-48f4-a66f-775e2ebe751d","Type":"ContainerDied","Data":"2acfe388416e6b6fb3e9ba1e8c0569689a277f26f9f65cd625c480052ec0deea"} Oct 30 19:57:47.439494 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:47.439134 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e67696-8f8b6cbd6-nxb2b" event={"ID":"c34c4b57-da35-48f4-a66f-775e2ebe751d","Type":"ContainerDied","Data":"ce966a062d5132bd33ea27421b13d3a9fc705cc1c3d64fa66f8bd4c82ff266bb"} Oct 30 19:57:47.439494 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:47.439156 4292 scope.go:122] "RemoveContainer" containerID="2acfe388416e6b6fb3e9ba1e8c0569689a277f26f9f65cd625c480052ec0deea" Oct 30 19:57:47.440555 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:47.440532 4292 generic.go:356] "Generic (PLEG): container finished" podID="68c94745-a215-46df-bdba-d4aa4fd54f0e" containerID="5dfa845923609f3fc71bb0104b1a72d64b8f127d5d72fb0ded04628b22398577" exitCode=0 Oct 30 19:57:47.440619 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:47.440561 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e67696-66c9b6dfdc-g5mtr" event={"ID":"68c94745-a215-46df-bdba-d4aa4fd54f0e","Type":"ContainerDied","Data":"5dfa845923609f3fc71bb0104b1a72d64b8f127d5d72fb0ded04628b22398577"} Oct 30 19:57:47.440619 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:47.440579 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e67696-66c9b6dfdc-g5mtr" Oct 30 19:57:47.440619 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:47.440595 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e67696-66c9b6dfdc-g5mtr" event={"ID":"68c94745-a215-46df-bdba-d4aa4fd54f0e","Type":"ContainerDied","Data":"937132bb6f487b1fed63480a6821c394d036eca2041a4621c10076f94ea4d2d6"} Oct 30 19:57:47.449661 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:47.449645 4292 scope.go:122] "RemoveContainer" containerID="2acfe388416e6b6fb3e9ba1e8c0569689a277f26f9f65cd625c480052ec0deea" Oct 30 19:57:47.449969 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:57:47.449951 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2acfe388416e6b6fb3e9ba1e8c0569689a277f26f9f65cd625c480052ec0deea\": container with ID starting with 2acfe388416e6b6fb3e9ba1e8c0569689a277f26f9f65cd625c480052ec0deea not found: ID does not exist" containerID="2acfe388416e6b6fb3e9ba1e8c0569689a277f26f9f65cd625c480052ec0deea" Oct 30 19:57:47.450026 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:47.449977 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2acfe388416e6b6fb3e9ba1e8c0569689a277f26f9f65cd625c480052ec0deea"} err="failed to get container status \"2acfe388416e6b6fb3e9ba1e8c0569689a277f26f9f65cd625c480052ec0deea\": rpc error: code = NotFound desc = could not find container \"2acfe388416e6b6fb3e9ba1e8c0569689a277f26f9f65cd625c480052ec0deea\": container with ID starting with 2acfe388416e6b6fb3e9ba1e8c0569689a277f26f9f65cd625c480052ec0deea not found: ID does not exist" Oct 30 19:57:47.450026 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:47.449993 4292 scope.go:122] "RemoveContainer" containerID="5dfa845923609f3fc71bb0104b1a72d64b8f127d5d72fb0ded04628b22398577" Oct 30 19:57:47.458154 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:47.458135 4292 scope.go:122] "RemoveContainer" containerID="5dfa845923609f3fc71bb0104b1a72d64b8f127d5d72fb0ded04628b22398577" Oct 30 19:57:47.458439 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:57:47.458416 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dfa845923609f3fc71bb0104b1a72d64b8f127d5d72fb0ded04628b22398577\": container with ID starting with 5dfa845923609f3fc71bb0104b1a72d64b8f127d5d72fb0ded04628b22398577 not found: ID does not exist" containerID="5dfa845923609f3fc71bb0104b1a72d64b8f127d5d72fb0ded04628b22398577" Oct 30 19:57:47.458523 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:47.458444 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dfa845923609f3fc71bb0104b1a72d64b8f127d5d72fb0ded04628b22398577"} err="failed to get container status \"5dfa845923609f3fc71bb0104b1a72d64b8f127d5d72fb0ded04628b22398577\": rpc error: code = NotFound desc = could not find container \"5dfa845923609f3fc71bb0104b1a72d64b8f127d5d72fb0ded04628b22398577\": container with ID starting with 5dfa845923609f3fc71bb0104b1a72d64b8f127d5d72fb0ded04628b22398577 not found: ID does not exist" Oct 30 19:57:47.614435 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:47.614400 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e67696-8f8b6cbd6-nxb2b"] Oct 30 19:57:47.665487 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:47.665411 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e67696-8f8b6cbd6-nxb2b"] Oct 30 19:57:47.730927 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:47.730894 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e67696-66c9b6dfdc-g5mtr"] Oct 30 19:57:47.733597 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:47.733575 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e67696-66c9b6dfdc-g5mtr"] Oct 30 19:57:48.583089 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:48.583059 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68c94745-a215-46df-bdba-d4aa4fd54f0e" path="/var/lib/kubelet/pods/68c94745-a215-46df-bdba-d4aa4fd54f0e/volumes" Oct 30 19:57:48.583507 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:48.583497 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c34c4b57-da35-48f4-a66f-775e2ebe751d" path="/var/lib/kubelet/pods/c34c4b57-da35-48f4-a66f-775e2ebe751d/volumes" Oct 30 19:57:54.494530 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:54.494501 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9"] Oct 30 19:57:54.494966 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:54.494707 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9" podUID="5bbd09dd-3324-4220-ad59-381beadfdf38" containerName="istio-proxy" containerID="cri-o://a3cdc21e06f9a2f9218e2ffdd61e7e2b50e93bc9abff5c479de3a4f32b8e0f67" gracePeriod=30 Oct 30 19:57:59.745186 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:59.745162 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9" Oct 30 19:57:59.823870 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:59.823844 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5bbd09dd-3324-4220-ad59-381beadfdf38-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bbd09dd-3324-4220-ad59-381beadfdf38-istio-podinfo\") pod \"5bbd09dd-3324-4220-ad59-381beadfdf38\" (UID: \"5bbd09dd-3324-4220-ad59-381beadfdf38\") " Oct 30 19:57:59.824018 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:59.823924 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bbd09dd-3324-4220-ad59-381beadfdf38-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bbd09dd-3324-4220-ad59-381beadfdf38-workload-socket\") pod \"5bbd09dd-3324-4220-ad59-381beadfdf38\" (UID: \"5bbd09dd-3324-4220-ad59-381beadfdf38\") " Oct 30 19:57:59.824109 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:59.824023 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bbd09dd-3324-4220-ad59-381beadfdf38-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bbd09dd-3324-4220-ad59-381beadfdf38-credential-socket\") pod \"5bbd09dd-3324-4220-ad59-381beadfdf38\" (UID: \"5bbd09dd-3324-4220-ad59-381beadfdf38\") " Oct 30 19:57:59.824109 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:59.824084 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bbd09dd-3324-4220-ad59-381beadfdf38-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bbd09dd-3324-4220-ad59-381beadfdf38-istio-data\") pod \"5bbd09dd-3324-4220-ad59-381beadfdf38\" (UID: \"5bbd09dd-3324-4220-ad59-381beadfdf38\") " Oct 30 19:57:59.824194 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:59.824145 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bbd09dd-3324-4220-ad59-381beadfdf38-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bbd09dd-3324-4220-ad59-381beadfdf38-workload-certs\") pod \"5bbd09dd-3324-4220-ad59-381beadfdf38\" (UID: \"5bbd09dd-3324-4220-ad59-381beadfdf38\") " Oct 30 19:57:59.824236 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:59.824190 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bbd09dd-3324-4220-ad59-381beadfdf38-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bbd09dd-3324-4220-ad59-381beadfdf38-istiod-ca-cert\") pod \"5bbd09dd-3324-4220-ad59-381beadfdf38\" (UID: \"5bbd09dd-3324-4220-ad59-381beadfdf38\") " Oct 30 19:57:59.824301 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:59.824244 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bbd09dd-3324-4220-ad59-381beadfdf38-workload-socket" pod "5bbd09dd-3324-4220-ad59-381beadfdf38" (UID: "5bbd09dd-3324-4220-ad59-381beadfdf38"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:57:59.824301 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:59.824255 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bbd09dd-3324-4220-ad59-381beadfdf38-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bbd09dd-3324-4220-ad59-381beadfdf38-istio-envoy\") pod \"5bbd09dd-3324-4220-ad59-381beadfdf38\" (UID: \"5bbd09dd-3324-4220-ad59-381beadfdf38\") " Oct 30 19:57:59.824301 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:59.824272 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bbd09dd-3324-4220-ad59-381beadfdf38-credential-socket" pod "5bbd09dd-3324-4220-ad59-381beadfdf38" (UID: "5bbd09dd-3324-4220-ad59-381beadfdf38"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:57:59.824469 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:59.824394 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bbd09dd-3324-4220-ad59-381beadfdf38-istio-token\" (UniqueName: \"kubernetes.io/projected/5bbd09dd-3324-4220-ad59-381beadfdf38-istio-token\") pod \"5bbd09dd-3324-4220-ad59-381beadfdf38\" (UID: \"5bbd09dd-3324-4220-ad59-381beadfdf38\") " Oct 30 19:57:59.824533 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:59.824445 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bbd09dd-3324-4220-ad59-381beadfdf38-istio-data" pod "5bbd09dd-3324-4220-ad59-381beadfdf38" (UID: "5bbd09dd-3324-4220-ad59-381beadfdf38"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:57:59.824533 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:59.824480 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bbd09dd-3324-4220-ad59-381beadfdf38-workload-certs" pod "5bbd09dd-3324-4220-ad59-381beadfdf38" (UID: "5bbd09dd-3324-4220-ad59-381beadfdf38"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:57:59.824533 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:59.824481 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bbd09dd-3324-4220-ad59-381beadfdf38-kube-api-access-28mk2\" (UniqueName: \"kubernetes.io/projected/5bbd09dd-3324-4220-ad59-381beadfdf38-kube-api-access-28mk2\") pod \"5bbd09dd-3324-4220-ad59-381beadfdf38\" (UID: \"5bbd09dd-3324-4220-ad59-381beadfdf38\") " Oct 30 19:57:59.824687 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:59.824609 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bbd09dd-3324-4220-ad59-381beadfdf38-istiod-ca-cert" pod "5bbd09dd-3324-4220-ad59-381beadfdf38" (UID: "5bbd09dd-3324-4220-ad59-381beadfdf38"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:57:59.825035 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:59.825010 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bbd09dd-3324-4220-ad59-381beadfdf38-workload-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:59.825131 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:59.825041 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bbd09dd-3324-4220-ad59-381beadfdf38-credential-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:59.825131 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:59.825059 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bbd09dd-3324-4220-ad59-381beadfdf38-istio-data\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:59.825131 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:59.825074 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bbd09dd-3324-4220-ad59-381beadfdf38-workload-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:59.825131 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:59.825089 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bbd09dd-3324-4220-ad59-381beadfdf38-istiod-ca-cert\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:59.826188 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:59.826165 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5bbd09dd-3324-4220-ad59-381beadfdf38-istio-podinfo" pod "5bbd09dd-3324-4220-ad59-381beadfdf38" (UID: "5bbd09dd-3324-4220-ad59-381beadfdf38"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:57:59.826324 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:59.826307 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bbd09dd-3324-4220-ad59-381beadfdf38-istio-envoy" pod "5bbd09dd-3324-4220-ad59-381beadfdf38" (UID: "5bbd09dd-3324-4220-ad59-381beadfdf38"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:57:59.826575 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:59.826554 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bbd09dd-3324-4220-ad59-381beadfdf38-kube-api-access-28mk2" pod "5bbd09dd-3324-4220-ad59-381beadfdf38" (UID: "5bbd09dd-3324-4220-ad59-381beadfdf38"). InnerVolumeSpecName "kube-api-access-28mk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:57:59.826647 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:59.826630 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bbd09dd-3324-4220-ad59-381beadfdf38-istio-token" pod "5bbd09dd-3324-4220-ad59-381beadfdf38" (UID: "5bbd09dd-3324-4220-ad59-381beadfdf38"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:57:59.926761 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:59.926726 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bbd09dd-3324-4220-ad59-381beadfdf38-istio-envoy\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:59.926761 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:59.926755 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bbd09dd-3324-4220-ad59-381beadfdf38-istio-token\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:59.926761 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:59.926766 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28mk2\" (UniqueName: \"kubernetes.io/projected/5bbd09dd-3324-4220-ad59-381beadfdf38-kube-api-access-28mk2\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:59.926992 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:57:59.926777 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bbd09dd-3324-4220-ad59-381beadfdf38-istio-podinfo\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:00.513389 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:00.513334 4292 generic.go:356] "Generic (PLEG): container finished" podID="5bbd09dd-3324-4220-ad59-381beadfdf38" containerID="a3cdc21e06f9a2f9218e2ffdd61e7e2b50e93bc9abff5c479de3a4f32b8e0f67" exitCode=0 Oct 30 19:58:00.513565 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:00.513409 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9" Oct 30 19:58:00.513565 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:00.513423 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9" event={"ID":"5bbd09dd-3324-4220-ad59-381beadfdf38","Type":"ContainerDied","Data":"a3cdc21e06f9a2f9218e2ffdd61e7e2b50e93bc9abff5c479de3a4f32b8e0f67"} Oct 30 19:58:00.513565 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:00.513518 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9" event={"ID":"5bbd09dd-3324-4220-ad59-381beadfdf38","Type":"ContainerDied","Data":"c23bfb4394e148ac97d705ce7546bb6d9099ffeddfb77f93e61b7161c2700cab"} Oct 30 19:58:00.513565 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:00.513543 4292 scope.go:122] "RemoveContainer" containerID="a3cdc21e06f9a2f9218e2ffdd61e7e2b50e93bc9abff5c479de3a4f32b8e0f67" Oct 30 19:58:00.523109 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:00.523089 4292 scope.go:122] "RemoveContainer" containerID="a3cdc21e06f9a2f9218e2ffdd61e7e2b50e93bc9abff5c479de3a4f32b8e0f67" Oct 30 19:58:00.523404 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:58:00.523381 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3cdc21e06f9a2f9218e2ffdd61e7e2b50e93bc9abff5c479de3a4f32b8e0f67\": container with ID starting with a3cdc21e06f9a2f9218e2ffdd61e7e2b50e93bc9abff5c479de3a4f32b8e0f67 not found: ID does not exist" containerID="a3cdc21e06f9a2f9218e2ffdd61e7e2b50e93bc9abff5c479de3a4f32b8e0f67" Oct 30 19:58:00.523473 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:00.523419 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3cdc21e06f9a2f9218e2ffdd61e7e2b50e93bc9abff5c479de3a4f32b8e0f67"} err="failed to get container status \"a3cdc21e06f9a2f9218e2ffdd61e7e2b50e93bc9abff5c479de3a4f32b8e0f67\": rpc error: code = NotFound desc = could not find container \"a3cdc21e06f9a2f9218e2ffdd61e7e2b50e93bc9abff5c479de3a4f32b8e0f67\": container with ID starting with a3cdc21e06f9a2f9218e2ffdd61e7e2b50e93bc9abff5c479de3a4f32b8e0f67 not found: ID does not exist" Oct 30 19:58:00.734702 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:00.734669 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9"] Oct 30 19:58:00.784898 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:00.784866 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5jlm9"] Oct 30 19:58:02.582837 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:02.582803 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bbd09dd-3324-4220-ad59-381beadfdf38" path="/var/lib/kubelet/pods/5bbd09dd-3324-4220-ad59-381beadfdf38/volumes" Oct 30 19:58:13.239393 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:58:13.239332 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:58:35.639959 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:35.639915 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c"] Oct 30 19:58:35.640371 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:35.640188 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c" podUID="02e3cbb1-bde4-4a15-b543-db2b4b820f99" containerName="istio-proxy" containerID="cri-o://0c45484666f573a2a3ef6aac32d0a55a797e6b40a1ebc4eb22af939746ab332b" gracePeriod=30 Oct 30 19:58:39.090590 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:39.090550 4292 patch_prober.go:29] interesting pod/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:58:39.091024 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:39.090650 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c" podUID="02e3cbb1-bde4-4a15-b543-db2b4b820f99" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:58:39.958690 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:39.958655 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lfxgm"] Oct 30 19:58:39.959136 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:39.959123 4292 cpu_manager.go:414] "containerMap: removing container" podUID="30db642e-9138-4699-8ffe-7f530f1e956e" containerName="maas-api" Oct 30 19:58:39.959193 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:39.959138 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30db642e-9138-4699-8ffe-7f530f1e956e" containerName="maas-api" Oct 30 19:58:39.959193 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:39.959147 4292 cpu_manager.go:414] "containerMap: removing container" podUID="3729f9f3-1ad0-48c2-a342-495c57602e6f" containerName="payload-pre-processing" Oct 30 19:58:39.959193 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:39.959152 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3729f9f3-1ad0-48c2-a342-495c57602e6f" containerName="payload-pre-processing" Oct 30 19:58:39.959193 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:39.959171 4292 cpu_manager.go:414] "containerMap: removing container" podUID="d7eb2b8c-8309-4951-82f7-e0f385da96a5" containerName="payload-processing" Oct 30 19:58:39.959193 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:39.959175 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7eb2b8c-8309-4951-82f7-e0f385da96a5" containerName="payload-processing" Oct 30 19:58:39.959396 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:39.959195 4292 cpu_manager.go:414] "containerMap: removing container" podUID="5bbd09dd-3324-4220-ad59-381beadfdf38" containerName="istio-proxy" Oct 30 19:58:39.959396 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:39.959201 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bbd09dd-3324-4220-ad59-381beadfdf38" containerName="istio-proxy" Oct 30 19:58:39.959396 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:39.959209 4292 cpu_manager.go:414] "containerMap: removing container" podUID="c34c4b57-da35-48f4-a66f-775e2ebe751d" containerName="payload-processing" Oct 30 19:58:39.959396 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:39.959216 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c34c4b57-da35-48f4-a66f-775e2ebe751d" containerName="payload-processing" Oct 30 19:58:39.959396 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:39.959237 4292 cpu_manager.go:414] "containerMap: removing container" podUID="2bb658ce-897b-46f3-8b91-97c611f6221d" containerName="payload-pre-processing" Oct 30 19:58:39.959396 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:39.959242 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bb658ce-897b-46f3-8b91-97c611f6221d" containerName="payload-pre-processing" Oct 30 19:58:39.959396 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:39.959248 4292 cpu_manager.go:414] "containerMap: removing container" podUID="fb0e159a-7596-4936-989d-0cc0cdc952af" containerName="maas-api" Oct 30 19:58:39.959396 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:39.959253 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb0e159a-7596-4936-989d-0cc0cdc952af" containerName="maas-api" Oct 30 19:58:39.959396 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:39.959261 4292 cpu_manager.go:414] "containerMap: removing container" podUID="68c94745-a215-46df-bdba-d4aa4fd54f0e" containerName="payload-pre-processing" Oct 30 19:58:39.959396 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:39.959266 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68c94745-a215-46df-bdba-d4aa4fd54f0e" containerName="payload-pre-processing" Oct 30 19:58:39.959396 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:39.959272 4292 cpu_manager.go:414] "containerMap: removing container" podUID="730c37c8-3814-42cc-b40e-c557279ad8ca" containerName="payload-pre-processing" Oct 30 19:58:39.959396 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:39.959277 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="730c37c8-3814-42cc-b40e-c557279ad8ca" containerName="payload-pre-processing" Oct 30 19:58:39.959396 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:39.959283 4292 cpu_manager.go:414] "containerMap: removing container" podUID="caf34ded-76a9-4af9-9b38-bd5437157111" containerName="istio-proxy" Oct 30 19:58:39.959396 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:39.959289 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caf34ded-76a9-4af9-9b38-bd5437157111" containerName="istio-proxy" Oct 30 19:58:39.959396 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:39.959295 4292 cpu_manager.go:414] "containerMap: removing container" podUID="a1f916af-267b-4057-a8aa-eef309537220" containerName="payload-processing" Oct 30 19:58:39.959396 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:39.959299 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1f916af-267b-4057-a8aa-eef309537220" containerName="payload-processing" Oct 30 19:58:39.959396 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:39.959307 4292 cpu_manager.go:414] "containerMap: removing container" podUID="dd0da58c-2f5a-4733-bea2-d2599a9931f2" containerName="istio-proxy" Oct 30 19:58:39.959396 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:39.959312 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd0da58c-2f5a-4733-bea2-d2599a9931f2" containerName="istio-proxy" Oct 30 19:58:39.959396 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:39.959317 4292 cpu_manager.go:414] "containerMap: removing container" podUID="56bef2d5-12bf-4ccb-8dea-f031595ddd96" containerName="payload-processing" Oct 30 19:58:39.959396 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:39.959321 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56bef2d5-12bf-4ccb-8dea-f031595ddd96" containerName="payload-processing" Oct 30 19:58:39.959396 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:39.959334 4292 cpu_manager.go:414] "containerMap: removing container" podUID="317c18b6-5b6b-4af5-9ba3-3d5ad719d53b" containerName="istio-proxy" Oct 30 19:58:39.959396 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:39.959338 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="317c18b6-5b6b-4af5-9ba3-3d5ad719d53b" containerName="istio-proxy" Oct 30 19:58:39.959396 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:39.959375 4292 cpu_manager.go:414] "containerMap: removing container" podUID="aa4437a1-8344-44c8-91a9-6ae667cbad68" containerName="payload-pre-processing" Oct 30 19:58:39.959396 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:39.959380 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa4437a1-8344-44c8-91a9-6ae667cbad68" containerName="payload-pre-processing" Oct 30 19:58:39.960137 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:39.959454 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7eb2b8c-8309-4951-82f7-e0f385da96a5" containerName="payload-processing" Oct 30 19:58:39.960137 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:39.959464 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bbd09dd-3324-4220-ad59-381beadfdf38" containerName="istio-proxy" Oct 30 19:58:39.960137 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:39.959470 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="c34c4b57-da35-48f4-a66f-775e2ebe751d" containerName="payload-processing" Oct 30 19:58:39.960137 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:39.959478 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bb658ce-897b-46f3-8b91-97c611f6221d" containerName="payload-pre-processing" Oct 30 19:58:39.960137 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:39.959483 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb0e159a-7596-4936-989d-0cc0cdc952af" containerName="maas-api" Oct 30 19:58:39.960137 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:39.959488 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="68c94745-a215-46df-bdba-d4aa4fd54f0e" containerName="payload-pre-processing" Oct 30 19:58:39.960137 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:39.959494 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="730c37c8-3814-42cc-b40e-c557279ad8ca" containerName="payload-pre-processing" Oct 30 19:58:39.960137 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:39.959499 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="caf34ded-76a9-4af9-9b38-bd5437157111" containerName="istio-proxy" Oct 30 19:58:39.960137 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:39.959504 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1f916af-267b-4057-a8aa-eef309537220" containerName="payload-processing" Oct 30 19:58:39.960137 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:39.959509 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd0da58c-2f5a-4733-bea2-d2599a9931f2" containerName="istio-proxy" Oct 30 19:58:39.960137 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:39.959514 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="56bef2d5-12bf-4ccb-8dea-f031595ddd96" containerName="payload-processing" Oct 30 19:58:39.960137 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:39.959520 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="317c18b6-5b6b-4af5-9ba3-3d5ad719d53b" containerName="istio-proxy" Oct 30 19:58:39.960137 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:39.959527 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa4437a1-8344-44c8-91a9-6ae667cbad68" containerName="payload-pre-processing" Oct 30 19:58:39.960137 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:39.959532 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="30db642e-9138-4699-8ffe-7f530f1e956e" containerName="maas-api" Oct 30 19:58:39.960137 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:39.959536 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="3729f9f3-1ad0-48c2-a342-495c57602e6f" containerName="payload-pre-processing" Oct 30 19:58:39.962884 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:39.962869 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lfxgm" Oct 30 19:58:40.064542 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.064456 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac6a02b1-f1ae-4bce-a1ac-b925820fa432-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-lfxgm\" (UID: \"ac6a02b1-f1ae-4bce-a1ac-b925820fa432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lfxgm" Oct 30 19:58:40.064542 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.064527 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac6a02b1-f1ae-4bce-a1ac-b925820fa432-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-lfxgm\" (UID: \"ac6a02b1-f1ae-4bce-a1ac-b925820fa432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lfxgm" Oct 30 19:58:40.064819 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.064793 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8xws\" (UniqueName: \"kubernetes.io/projected/ac6a02b1-f1ae-4bce-a1ac-b925820fa432-kube-api-access-h8xws\") pod \"maas-default-gateway-openshift-default-694db55c87-lfxgm\" (UID: \"ac6a02b1-f1ae-4bce-a1ac-b925820fa432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lfxgm" Oct 30 19:58:40.064946 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.064875 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac6a02b1-f1ae-4bce-a1ac-b925820fa432-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-lfxgm\" (UID: \"ac6a02b1-f1ae-4bce-a1ac-b925820fa432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lfxgm" Oct 30 19:58:40.065014 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.064945 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac6a02b1-f1ae-4bce-a1ac-b925820fa432-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-lfxgm\" (UID: \"ac6a02b1-f1ae-4bce-a1ac-b925820fa432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lfxgm" Oct 30 19:58:40.065014 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.064972 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac6a02b1-f1ae-4bce-a1ac-b925820fa432-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-lfxgm\" (UID: \"ac6a02b1-f1ae-4bce-a1ac-b925820fa432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lfxgm" Oct 30 19:58:40.065093 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.065034 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac6a02b1-f1ae-4bce-a1ac-b925820fa432-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-lfxgm\" (UID: \"ac6a02b1-f1ae-4bce-a1ac-b925820fa432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lfxgm" Oct 30 19:58:40.065128 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.065102 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac6a02b1-f1ae-4bce-a1ac-b925820fa432-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-lfxgm\" (UID: \"ac6a02b1-f1ae-4bce-a1ac-b925820fa432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lfxgm" Oct 30 19:58:40.065164 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.065128 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac6a02b1-f1ae-4bce-a1ac-b925820fa432-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-lfxgm\" (UID: \"ac6a02b1-f1ae-4bce-a1ac-b925820fa432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lfxgm" Oct 30 19:58:40.086731 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.086699 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lfxgm"] Oct 30 19:58:40.166449 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.166413 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac6a02b1-f1ae-4bce-a1ac-b925820fa432-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-lfxgm\" (UID: \"ac6a02b1-f1ae-4bce-a1ac-b925820fa432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lfxgm" Oct 30 19:58:40.166919 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.166518 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac6a02b1-f1ae-4bce-a1ac-b925820fa432-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-lfxgm\" (UID: \"ac6a02b1-f1ae-4bce-a1ac-b925820fa432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lfxgm" Oct 30 19:58:40.166919 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.166560 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac6a02b1-f1ae-4bce-a1ac-b925820fa432-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-lfxgm\" (UID: \"ac6a02b1-f1ae-4bce-a1ac-b925820fa432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lfxgm" Oct 30 19:58:40.166919 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.166628 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac6a02b1-f1ae-4bce-a1ac-b925820fa432-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-lfxgm\" (UID: \"ac6a02b1-f1ae-4bce-a1ac-b925820fa432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lfxgm" Oct 30 19:58:40.166919 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.166683 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac6a02b1-f1ae-4bce-a1ac-b925820fa432-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-lfxgm\" (UID: \"ac6a02b1-f1ae-4bce-a1ac-b925820fa432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lfxgm" Oct 30 19:58:40.166919 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.166738 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8xws\" (UniqueName: \"kubernetes.io/projected/ac6a02b1-f1ae-4bce-a1ac-b925820fa432-kube-api-access-h8xws\") pod \"maas-default-gateway-openshift-default-694db55c87-lfxgm\" (UID: \"ac6a02b1-f1ae-4bce-a1ac-b925820fa432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lfxgm" Oct 30 19:58:40.166919 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.166772 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac6a02b1-f1ae-4bce-a1ac-b925820fa432-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-lfxgm\" (UID: \"ac6a02b1-f1ae-4bce-a1ac-b925820fa432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lfxgm" Oct 30 19:58:40.166919 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.166814 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac6a02b1-f1ae-4bce-a1ac-b925820fa432-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-lfxgm\" (UID: \"ac6a02b1-f1ae-4bce-a1ac-b925820fa432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lfxgm" Oct 30 19:58:40.166919 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.166836 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac6a02b1-f1ae-4bce-a1ac-b925820fa432-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-lfxgm\" (UID: \"ac6a02b1-f1ae-4bce-a1ac-b925820fa432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lfxgm" Oct 30 19:58:40.166919 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.166840 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac6a02b1-f1ae-4bce-a1ac-b925820fa432-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-lfxgm\" (UID: \"ac6a02b1-f1ae-4bce-a1ac-b925820fa432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lfxgm" Oct 30 19:58:40.167406 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.167013 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac6a02b1-f1ae-4bce-a1ac-b925820fa432-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-lfxgm\" (UID: \"ac6a02b1-f1ae-4bce-a1ac-b925820fa432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lfxgm" Oct 30 19:58:40.167406 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.167211 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac6a02b1-f1ae-4bce-a1ac-b925820fa432-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-lfxgm\" (UID: \"ac6a02b1-f1ae-4bce-a1ac-b925820fa432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lfxgm" Oct 30 19:58:40.167406 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.167276 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac6a02b1-f1ae-4bce-a1ac-b925820fa432-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-lfxgm\" (UID: \"ac6a02b1-f1ae-4bce-a1ac-b925820fa432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lfxgm" Oct 30 19:58:40.167551 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.167507 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac6a02b1-f1ae-4bce-a1ac-b925820fa432-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-lfxgm\" (UID: \"ac6a02b1-f1ae-4bce-a1ac-b925820fa432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lfxgm" Oct 30 19:58:40.169400 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.169372 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac6a02b1-f1ae-4bce-a1ac-b925820fa432-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-lfxgm\" (UID: \"ac6a02b1-f1ae-4bce-a1ac-b925820fa432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lfxgm" Oct 30 19:58:40.169727 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.169706 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac6a02b1-f1ae-4bce-a1ac-b925820fa432-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-lfxgm\" (UID: \"ac6a02b1-f1ae-4bce-a1ac-b925820fa432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lfxgm" Oct 30 19:58:40.232719 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.232630 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8xws\" (UniqueName: \"kubernetes.io/projected/ac6a02b1-f1ae-4bce-a1ac-b925820fa432-kube-api-access-h8xws\") pod \"maas-default-gateway-openshift-default-694db55c87-lfxgm\" (UID: \"ac6a02b1-f1ae-4bce-a1ac-b925820fa432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lfxgm" Oct 30 19:58:40.232932 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.232778 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac6a02b1-f1ae-4bce-a1ac-b925820fa432-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-lfxgm\" (UID: \"ac6a02b1-f1ae-4bce-a1ac-b925820fa432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lfxgm" Oct 30 19:58:40.276275 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.276249 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lfxgm" Oct 30 19:58:40.720078 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.720049 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lfxgm"] Oct 30 19:58:40.720585 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:58:40.720561 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac6a02b1_f1ae_4bce_a1ac_b925820fa432.slice/crio-5fe8a2a4e479e429049d4c15399bf19002e526ead7b559f1049d2e000c5d8f1c WatchSource:0}: Error finding container 5fe8a2a4e479e429049d4c15399bf19002e526ead7b559f1049d2e000c5d8f1c: Status 404 returned error can't find the container with id 5fe8a2a4e479e429049d4c15399bf19002e526ead7b559f1049d2e000c5d8f1c Oct 30 19:58:40.746407 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.746368 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:58:40.746499 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.746470 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:58:40.746536 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.746523 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:58:40.896734 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.896702 4292 generic.go:356] "Generic (PLEG): container finished" podID="02e3cbb1-bde4-4a15-b543-db2b4b820f99" containerID="0c45484666f573a2a3ef6aac32d0a55a797e6b40a1ebc4eb22af939746ab332b" exitCode=0 Oct 30 19:58:40.896891 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.896784 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c" event={"ID":"02e3cbb1-bde4-4a15-b543-db2b4b820f99","Type":"ContainerDied","Data":"0c45484666f573a2a3ef6aac32d0a55a797e6b40a1ebc4eb22af939746ab332b"} Oct 30 19:58:40.896891 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.896880 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c" event={"ID":"02e3cbb1-bde4-4a15-b543-db2b4b820f99","Type":"ContainerDied","Data":"a5ece29fba64bffd1176e01abe0697fa09ad023a455b83c0535063fc5ac45e28"} Oct 30 19:58:40.896980 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.896903 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5ece29fba64bffd1176e01abe0697fa09ad023a455b83c0535063fc5ac45e28" Oct 30 19:58:40.896980 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.896923 4292 scope.go:122] "RemoveContainer" containerID="28ea1d303ec7b4268abcd0b79525c9c6173ea784f25e5c8364afd72e9e299319" Oct 30 19:58:40.898200 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.898180 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c" Oct 30 19:58:40.898749 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.898722 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lfxgm" event={"ID":"ac6a02b1-f1ae-4bce-a1ac-b925820fa432","Type":"ContainerStarted","Data":"94bce6f54a19ee4acc178c036b1e49d5a72ddc3c50be252d911e0fab61038846"} Oct 30 19:58:40.898842 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.898779 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lfxgm" event={"ID":"ac6a02b1-f1ae-4bce-a1ac-b925820fa432","Type":"ContainerStarted","Data":"5fe8a2a4e479e429049d4c15399bf19002e526ead7b559f1049d2e000c5d8f1c"} Oct 30 19:58:40.983301 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.983270 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02e3cbb1-bde4-4a15-b543-db2b4b820f99-kube-api-access-bcwz6\" (UniqueName: \"kubernetes.io/projected/02e3cbb1-bde4-4a15-b543-db2b4b820f99-kube-api-access-bcwz6\") pod \"02e3cbb1-bde4-4a15-b543-db2b4b820f99\" (UID: \"02e3cbb1-bde4-4a15-b543-db2b4b820f99\") " Oct 30 19:58:40.983511 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.983463 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02e3cbb1-bde4-4a15-b543-db2b4b820f99-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02e3cbb1-bde4-4a15-b543-db2b4b820f99-workload-socket\") pod \"02e3cbb1-bde4-4a15-b543-db2b4b820f99\" (UID: \"02e3cbb1-bde4-4a15-b543-db2b4b820f99\") " Oct 30 19:58:40.983584 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.983518 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02e3cbb1-bde4-4a15-b543-db2b4b820f99-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02e3cbb1-bde4-4a15-b543-db2b4b820f99-istio-envoy\") pod \"02e3cbb1-bde4-4a15-b543-db2b4b820f99\" (UID: \"02e3cbb1-bde4-4a15-b543-db2b4b820f99\") " Oct 30 19:58:40.983584 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.983569 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02e3cbb1-bde4-4a15-b543-db2b4b820f99-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02e3cbb1-bde4-4a15-b543-db2b4b820f99-credential-socket\") pod \"02e3cbb1-bde4-4a15-b543-db2b4b820f99\" (UID: \"02e3cbb1-bde4-4a15-b543-db2b4b820f99\") " Oct 30 19:58:40.983726 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.983620 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02e3cbb1-bde4-4a15-b543-db2b4b820f99-istio-token\" (UniqueName: \"kubernetes.io/projected/02e3cbb1-bde4-4a15-b543-db2b4b820f99-istio-token\") pod \"02e3cbb1-bde4-4a15-b543-db2b4b820f99\" (UID: \"02e3cbb1-bde4-4a15-b543-db2b4b820f99\") " Oct 30 19:58:40.983726 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.983661 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02e3cbb1-bde4-4a15-b543-db2b4b820f99-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02e3cbb1-bde4-4a15-b543-db2b4b820f99-workload-certs\") pod \"02e3cbb1-bde4-4a15-b543-db2b4b820f99\" (UID: \"02e3cbb1-bde4-4a15-b543-db2b4b820f99\") " Oct 30 19:58:40.983726 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.983690 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02e3cbb1-bde4-4a15-b543-db2b4b820f99-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02e3cbb1-bde4-4a15-b543-db2b4b820f99-istio-data\") pod \"02e3cbb1-bde4-4a15-b543-db2b4b820f99\" (UID: \"02e3cbb1-bde4-4a15-b543-db2b4b820f99\") " Oct 30 19:58:40.983888 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.983777 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02e3cbb1-bde4-4a15-b543-db2b4b820f99-workload-socket" pod "02e3cbb1-bde4-4a15-b543-db2b4b820f99" (UID: "02e3cbb1-bde4-4a15-b543-db2b4b820f99"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:40.983950 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.983922 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02e3cbb1-bde4-4a15-b543-db2b4b820f99-credential-socket" pod "02e3cbb1-bde4-4a15-b543-db2b4b820f99" (UID: "02e3cbb1-bde4-4a15-b543-db2b4b820f99"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:40.984079 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.984027 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02e3cbb1-bde4-4a15-b543-db2b4b820f99-istio-data" pod "02e3cbb1-bde4-4a15-b543-db2b4b820f99" (UID: "02e3cbb1-bde4-4a15-b543-db2b4b820f99"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:40.984079 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.984072 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/02e3cbb1-bde4-4a15-b543-db2b4b820f99-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02e3cbb1-bde4-4a15-b543-db2b4b820f99-istio-podinfo\") pod \"02e3cbb1-bde4-4a15-b543-db2b4b820f99\" (UID: \"02e3cbb1-bde4-4a15-b543-db2b4b820f99\") " Oct 30 19:58:40.984079 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.984102 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02e3cbb1-bde4-4a15-b543-db2b4b820f99-workload-certs" pod "02e3cbb1-bde4-4a15-b543-db2b4b820f99" (UID: "02e3cbb1-bde4-4a15-b543-db2b4b820f99"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:40.984381 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.984122 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02e3cbb1-bde4-4a15-b543-db2b4b820f99-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02e3cbb1-bde4-4a15-b543-db2b4b820f99-istiod-ca-cert\") pod \"02e3cbb1-bde4-4a15-b543-db2b4b820f99\" (UID: \"02e3cbb1-bde4-4a15-b543-db2b4b820f99\") " Oct 30 19:58:40.984488 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.984465 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02e3cbb1-bde4-4a15-b543-db2b4b820f99-istiod-ca-cert" pod "02e3cbb1-bde4-4a15-b543-db2b4b820f99" (UID: "02e3cbb1-bde4-4a15-b543-db2b4b820f99"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:58:40.984881 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.984860 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02e3cbb1-bde4-4a15-b543-db2b4b820f99-workload-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:40.985008 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.984889 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02e3cbb1-bde4-4a15-b543-db2b4b820f99-credential-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:40.985008 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.984904 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02e3cbb1-bde4-4a15-b543-db2b4b820f99-workload-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:40.985008 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.984918 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02e3cbb1-bde4-4a15-b543-db2b4b820f99-istio-data\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:40.985008 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.984931 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02e3cbb1-bde4-4a15-b543-db2b4b820f99-istiod-ca-cert\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:40.985802 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.985776 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02e3cbb1-bde4-4a15-b543-db2b4b820f99-kube-api-access-bcwz6" pod "02e3cbb1-bde4-4a15-b543-db2b4b820f99" (UID: "02e3cbb1-bde4-4a15-b543-db2b4b820f99"). InnerVolumeSpecName "kube-api-access-bcwz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:40.986020 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.986000 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02e3cbb1-bde4-4a15-b543-db2b4b820f99-istio-token" pod "02e3cbb1-bde4-4a15-b543-db2b4b820f99" (UID: "02e3cbb1-bde4-4a15-b543-db2b4b820f99"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:40.986292 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.986241 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02e3cbb1-bde4-4a15-b543-db2b4b820f99-istio-envoy" pod "02e3cbb1-bde4-4a15-b543-db2b4b820f99" (UID: "02e3cbb1-bde4-4a15-b543-db2b4b820f99"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:40.986344 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:40.986277 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/02e3cbb1-bde4-4a15-b543-db2b4b820f99-istio-podinfo" pod "02e3cbb1-bde4-4a15-b543-db2b4b820f99" (UID: "02e3cbb1-bde4-4a15-b543-db2b4b820f99"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:58:41.086577 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:41.086544 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02e3cbb1-bde4-4a15-b543-db2b4b820f99-istio-podinfo\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:41.086577 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:41.086571 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcwz6\" (UniqueName: \"kubernetes.io/projected/02e3cbb1-bde4-4a15-b543-db2b4b820f99-kube-api-access-bcwz6\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:41.086577 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:41.086582 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02e3cbb1-bde4-4a15-b543-db2b4b820f99-istio-envoy\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:41.086806 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:41.086591 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02e3cbb1-bde4-4a15-b543-db2b4b820f99-istio-token\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:41.148038 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:41.147989 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lfxgm" podStartSLOduration=2.147975685 podStartE2EDuration="2.147975685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:41.146031323 +0000 UTC m=+2024.664346780" watchObservedRunningTime="2026-10-30 19:58:41.147975685 +0000 UTC m=+2024.666291142" Oct 30 19:58:41.277466 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:41.277433 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lfxgm" Oct 30 19:58:41.278854 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:41.278827 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-lfxgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 19:58:41.279007 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:41.278886 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lfxgm" podUID="ac6a02b1-f1ae-4bce-a1ac-b925820fa432" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 19:58:41.904878 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:41.904845 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c" Oct 30 19:58:42.207160 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:42.207080 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c"] Oct 30 19:58:42.212416 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:42.212393 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-69932e-openshift-default-6f5f5545b9-zcp9c"] Oct 30 19:58:42.276899 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:42.276857 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-lfxgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 19:58:42.277048 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:42.276938 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lfxgm" podUID="ac6a02b1-f1ae-4bce-a1ac-b925820fa432" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 19:58:42.584960 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:42.584929 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02e3cbb1-bde4-4a15-b543-db2b4b820f99" path="/var/lib/kubelet/pods/02e3cbb1-bde4-4a15-b543-db2b4b820f99/volumes" Oct 30 19:58:43.222751 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:58:43.222709 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:58:43.276872 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:43.276839 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-lfxgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 19:58:43.277052 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:43.276888 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lfxgm" podUID="ac6a02b1-f1ae-4bce-a1ac-b925820fa432" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 19:58:44.276951 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:44.276913 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-lfxgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 19:58:44.277392 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:44.276982 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lfxgm" podUID="ac6a02b1-f1ae-4bce-a1ac-b925820fa432" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 19:58:45.277058 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:45.276698 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-lfxgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 19:58:45.277058 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:45.276764 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lfxgm" podUID="ac6a02b1-f1ae-4bce-a1ac-b925820fa432" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 19:58:46.276917 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:46.276875 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-lfxgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 19:58:46.277106 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:46.276941 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lfxgm" podUID="ac6a02b1-f1ae-4bce-a1ac-b925820fa432" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 19:58:47.276647 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:47.276610 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-lfxgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 19:58:47.276832 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:47.276672 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lfxgm" podUID="ac6a02b1-f1ae-4bce-a1ac-b925820fa432" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 19:58:48.276839 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:48.276805 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-lfxgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 19:58:48.277256 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:48.276868 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lfxgm" podUID="ac6a02b1-f1ae-4bce-a1ac-b925820fa432" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 19:58:49.281337 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:49.281301 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lfxgm" Oct 30 19:58:49.281746 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:49.281600 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lfxgm" Oct 30 19:58:49.282232 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:49.282208 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lfxgm" Oct 30 19:58:54.187792 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:54.187753 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj"] Oct 30 19:58:54.188242 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:54.187955 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj" podUID="8efc6802-ecd2-45b1-af43-799ca4e9bcb1" containerName="istio-proxy" containerID="cri-o://993f26fffefb36c7206d53e299629013acae48dfa108f4fe48988446cc2c349f" gracePeriod=30 Oct 30 19:58:54.768069 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:54.768025 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f"] Oct 30 19:58:54.768300 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:54.768269 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f" podUID="3845b969-7d5b-48d4-a2d8-dfbe93f91c05" containerName="istio-proxy" containerID="cri-o://0470224b914e5efac94b13fd57065c9111b37a70a940942abe7b68b24b4dc94a" gracePeriod=30 Oct 30 19:58:58.513538 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:58.513494 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:58:58.513911 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:58.513563 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f" podUID="3845b969-7d5b-48d4-a2d8-dfbe93f91c05" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:58:59.345199 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:59.345154 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e67696-6796dcfcf6-zncj4"] Oct 30 19:58:59.345418 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:59.345378 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e67696-6796dcfcf6-zncj4" podUID="b6c13eb3-0acd-4a2e-9171-28042161aa6f" containerName="maas-api" containerID="cri-o://590b8e96b4ad68e9e46ffccfcc9e537d9fb1ad4d9be5ff138cc28a9785d99b86" gracePeriod=30 Oct 30 19:58:59.476957 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:59.476929 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj" Oct 30 19:58:59.631407 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:59.610717 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8efc6802-ecd2-45b1-af43-799ca4e9bcb1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8efc6802-ecd2-45b1-af43-799ca4e9bcb1-credential-socket\") pod \"8efc6802-ecd2-45b1-af43-799ca4e9bcb1\" (UID: \"8efc6802-ecd2-45b1-af43-799ca4e9bcb1\") " Oct 30 19:58:59.631407 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:59.610806 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8efc6802-ecd2-45b1-af43-799ca4e9bcb1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8efc6802-ecd2-45b1-af43-799ca4e9bcb1-workload-socket\") pod \"8efc6802-ecd2-45b1-af43-799ca4e9bcb1\" (UID: \"8efc6802-ecd2-45b1-af43-799ca4e9bcb1\") " Oct 30 19:58:59.631407 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:59.610852 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8efc6802-ecd2-45b1-af43-799ca4e9bcb1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8efc6802-ecd2-45b1-af43-799ca4e9bcb1-workload-certs\") pod \"8efc6802-ecd2-45b1-af43-799ca4e9bcb1\" (UID: \"8efc6802-ecd2-45b1-af43-799ca4e9bcb1\") " Oct 30 19:58:59.631407 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:59.610880 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8efc6802-ecd2-45b1-af43-799ca4e9bcb1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8efc6802-ecd2-45b1-af43-799ca4e9bcb1-istiod-ca-cert\") pod \"8efc6802-ecd2-45b1-af43-799ca4e9bcb1\" (UID: \"8efc6802-ecd2-45b1-af43-799ca4e9bcb1\") " Oct 30 19:58:59.631407 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:59.610947 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8efc6802-ecd2-45b1-af43-799ca4e9bcb1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8efc6802-ecd2-45b1-af43-799ca4e9bcb1-istio-envoy\") pod \"8efc6802-ecd2-45b1-af43-799ca4e9bcb1\" (UID: \"8efc6802-ecd2-45b1-af43-799ca4e9bcb1\") " Oct 30 19:58:59.631407 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:59.611000 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8efc6802-ecd2-45b1-af43-799ca4e9bcb1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8efc6802-ecd2-45b1-af43-799ca4e9bcb1-istio-podinfo\") pod \"8efc6802-ecd2-45b1-af43-799ca4e9bcb1\" (UID: \"8efc6802-ecd2-45b1-af43-799ca4e9bcb1\") " Oct 30 19:58:59.631407 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:59.611047 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8efc6802-ecd2-45b1-af43-799ca4e9bcb1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8efc6802-ecd2-45b1-af43-799ca4e9bcb1-istio-data\") pod \"8efc6802-ecd2-45b1-af43-799ca4e9bcb1\" (UID: \"8efc6802-ecd2-45b1-af43-799ca4e9bcb1\") " Oct 30 19:58:59.631407 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:59.611090 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8efc6802-ecd2-45b1-af43-799ca4e9bcb1-kube-api-access-zfp56\" (UniqueName: \"kubernetes.io/projected/8efc6802-ecd2-45b1-af43-799ca4e9bcb1-kube-api-access-zfp56\") pod \"8efc6802-ecd2-45b1-af43-799ca4e9bcb1\" (UID: \"8efc6802-ecd2-45b1-af43-799ca4e9bcb1\") " Oct 30 19:58:59.631407 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:59.611159 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8efc6802-ecd2-45b1-af43-799ca4e9bcb1-istio-token\" (UniqueName: \"kubernetes.io/projected/8efc6802-ecd2-45b1-af43-799ca4e9bcb1-istio-token\") pod \"8efc6802-ecd2-45b1-af43-799ca4e9bcb1\" (UID: \"8efc6802-ecd2-45b1-af43-799ca4e9bcb1\") " Oct 30 19:58:59.649374 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:59.642409 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8efc6802-ecd2-45b1-af43-799ca4e9bcb1-istiod-ca-cert" pod "8efc6802-ecd2-45b1-af43-799ca4e9bcb1" (UID: "8efc6802-ecd2-45b1-af43-799ca4e9bcb1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:58:59.649374 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:59.642762 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8efc6802-ecd2-45b1-af43-799ca4e9bcb1-credential-socket" pod "8efc6802-ecd2-45b1-af43-799ca4e9bcb1" (UID: "8efc6802-ecd2-45b1-af43-799ca4e9bcb1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:59.649374 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:59.642982 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8efc6802-ecd2-45b1-af43-799ca4e9bcb1-workload-socket" pod "8efc6802-ecd2-45b1-af43-799ca4e9bcb1" (UID: "8efc6802-ecd2-45b1-af43-799ca4e9bcb1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:59.649374 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:59.643193 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8efc6802-ecd2-45b1-af43-799ca4e9bcb1-workload-certs" pod "8efc6802-ecd2-45b1-af43-799ca4e9bcb1" (UID: "8efc6802-ecd2-45b1-af43-799ca4e9bcb1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:59.649374 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:59.643966 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8efc6802-ecd2-45b1-af43-799ca4e9bcb1-istio-data" pod "8efc6802-ecd2-45b1-af43-799ca4e9bcb1" (UID: "8efc6802-ecd2-45b1-af43-799ca4e9bcb1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:59.680563 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:59.679703 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8efc6802-ecd2-45b1-af43-799ca4e9bcb1-credential-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:59.680563 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:59.679737 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8efc6802-ecd2-45b1-af43-799ca4e9bcb1-workload-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:59.680563 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:59.679753 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8efc6802-ecd2-45b1-af43-799ca4e9bcb1-workload-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:59.680563 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:59.679767 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8efc6802-ecd2-45b1-af43-799ca4e9bcb1-istiod-ca-cert\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:59.680563 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:59.679781 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8efc6802-ecd2-45b1-af43-799ca4e9bcb1-istio-data\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:59.684830 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:59.684416 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7337-a890cc"] Oct 30 19:58:59.685024 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:59.684997 4292 cpu_manager.go:414] "containerMap: removing container" podUID="8efc6802-ecd2-45b1-af43-799ca4e9bcb1" containerName="istio-proxy" Oct 30 19:58:59.685024 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:59.685013 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8efc6802-ecd2-45b1-af43-799ca4e9bcb1" containerName="istio-proxy" Oct 30 19:58:59.685143 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:59.685034 4292 cpu_manager.go:414] "containerMap: removing container" podUID="02e3cbb1-bde4-4a15-b543-db2b4b820f99" containerName="istio-proxy" Oct 30 19:58:59.685143 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:59.685042 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02e3cbb1-bde4-4a15-b543-db2b4b820f99" containerName="istio-proxy" Oct 30 19:58:59.685252 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:59.685183 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="02e3cbb1-bde4-4a15-b543-db2b4b820f99" containerName="istio-proxy" Oct 30 19:58:59.685252 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:59.685201 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="8efc6802-ecd2-45b1-af43-799ca4e9bcb1" containerName="istio-proxy" Oct 30 19:58:59.687664 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:59.687445 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8efc6802-ecd2-45b1-af43-799ca4e9bcb1-istio-podinfo" pod "8efc6802-ecd2-45b1-af43-799ca4e9bcb1" (UID: "8efc6802-ecd2-45b1-af43-799ca4e9bcb1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:58:59.687664 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:59.687559 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8efc6802-ecd2-45b1-af43-799ca4e9bcb1-istio-envoy" pod "8efc6802-ecd2-45b1-af43-799ca4e9bcb1" (UID: "8efc6802-ecd2-45b1-af43-799ca4e9bcb1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:59.689321 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:59.688796 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8efc6802-ecd2-45b1-af43-799ca4e9bcb1-istio-token" pod "8efc6802-ecd2-45b1-af43-799ca4e9bcb1" (UID: "8efc6802-ecd2-45b1-af43-799ca4e9bcb1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:59.689321 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:59.689045 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8efc6802-ecd2-45b1-af43-799ca4e9bcb1-kube-api-access-zfp56" pod "8efc6802-ecd2-45b1-af43-799ca4e9bcb1" (UID: "8efc6802-ecd2-45b1-af43-799ca4e9bcb1"). InnerVolumeSpecName "kube-api-access-zfp56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:59.689632 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:59.689612 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-a890cc" Oct 30 19:58:59.691306 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:59.691238 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e67696-6796dcfcf6-zncj4" Oct 30 19:58:59.740406 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:59.740274 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7337-a890cc"] Oct 30 19:58:59.743996 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:59.743969 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-f4jr9\"" Oct 30 19:58:59.780441 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:59.780407 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6c13eb3-0acd-4a2e-9171-28042161aa6f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6c13eb3-0acd-4a2e-9171-28042161aa6f-metrics-certs\") pod \"b6c13eb3-0acd-4a2e-9171-28042161aa6f\" (UID: \"b6c13eb3-0acd-4a2e-9171-28042161aa6f\") " Oct 30 19:58:59.780622 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:59.780462 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6c13eb3-0acd-4a2e-9171-28042161aa6f-kube-api-access-67pps\" (UniqueName: \"kubernetes.io/projected/b6c13eb3-0acd-4a2e-9171-28042161aa6f-kube-api-access-67pps\") pod \"b6c13eb3-0acd-4a2e-9171-28042161aa6f\" (UID: \"b6c13eb3-0acd-4a2e-9171-28042161aa6f\") " Oct 30 19:58:59.780622 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:59.780534 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6c13eb3-0acd-4a2e-9171-28042161aa6f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6c13eb3-0acd-4a2e-9171-28042161aa6f-openshift-service-ca\") pod \"b6c13eb3-0acd-4a2e-9171-28042161aa6f\" (UID: \"b6c13eb3-0acd-4a2e-9171-28042161aa6f\") " Oct 30 19:58:59.780622 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:59.780604 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6c13eb3-0acd-4a2e-9171-28042161aa6f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6c13eb3-0acd-4a2e-9171-28042161aa6f-maas-api-tls\") pod \"b6c13eb3-0acd-4a2e-9171-28042161aa6f\" (UID: \"b6c13eb3-0acd-4a2e-9171-28042161aa6f\") " Oct 30 19:58:59.781125 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:59.780973 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5wm8\" (UniqueName: \"kubernetes.io/projected/189f3c86-4927-4f46-8634-b9c94c893452-kube-api-access-r5wm8\") pod \"test-curl-7337-a890cc\" (UID: \"189f3c86-4927-4f46-8634-b9c94c893452\") " pod="opendatahub/test-curl-7337-a890cc" Oct 30 19:58:59.781125 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:59.781084 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8efc6802-ecd2-45b1-af43-799ca4e9bcb1-istio-envoy\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:59.781125 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:59.781100 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8efc6802-ecd2-45b1-af43-799ca4e9bcb1-istio-podinfo\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:59.781125 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:59.781115 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfp56\" (UniqueName: \"kubernetes.io/projected/8efc6802-ecd2-45b1-af43-799ca4e9bcb1-kube-api-access-zfp56\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:59.781510 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:59.781131 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8efc6802-ecd2-45b1-af43-799ca4e9bcb1-istio-token\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:59.781572 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:59.781514 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6c13eb3-0acd-4a2e-9171-28042161aa6f-openshift-service-ca" pod "b6c13eb3-0acd-4a2e-9171-28042161aa6f" (UID: "b6c13eb3-0acd-4a2e-9171-28042161aa6f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:58:59.783244 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:59.783208 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6c13eb3-0acd-4a2e-9171-28042161aa6f-kube-api-access-67pps" pod "b6c13eb3-0acd-4a2e-9171-28042161aa6f" (UID: "b6c13eb3-0acd-4a2e-9171-28042161aa6f"). InnerVolumeSpecName "kube-api-access-67pps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:59.784133 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:59.784108 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6c13eb3-0acd-4a2e-9171-28042161aa6f-maas-api-tls" pod "b6c13eb3-0acd-4a2e-9171-28042161aa6f" (UID: "b6c13eb3-0acd-4a2e-9171-28042161aa6f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:58:59.784864 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:59.784845 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6c13eb3-0acd-4a2e-9171-28042161aa6f-metrics-certs" pod "b6c13eb3-0acd-4a2e-9171-28042161aa6f" (UID: "b6c13eb3-0acd-4a2e-9171-28042161aa6f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:58:59.885027 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:59.882547 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5wm8\" (UniqueName: \"kubernetes.io/projected/189f3c86-4927-4f46-8634-b9c94c893452-kube-api-access-r5wm8\") pod \"test-curl-7337-a890cc\" (UID: \"189f3c86-4927-4f46-8634-b9c94c893452\") " pod="opendatahub/test-curl-7337-a890cc" Oct 30 19:58:59.885027 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:59.882632 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6c13eb3-0acd-4a2e-9171-28042161aa6f-maas-api-tls\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:59.885027 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:59.882648 4292 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6c13eb3-0acd-4a2e-9171-28042161aa6f-metrics-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:59.885027 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:59.882664 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67pps\" (UniqueName: \"kubernetes.io/projected/b6c13eb3-0acd-4a2e-9171-28042161aa6f-kube-api-access-67pps\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:59.885027 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:59.882680 4292 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6c13eb3-0acd-4a2e-9171-28042161aa6f-openshift-service-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:59.946017 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:58:59.945982 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5wm8\" (UniqueName: \"kubernetes.io/projected/189f3c86-4927-4f46-8634-b9c94c893452-kube-api-access-r5wm8\") pod \"test-curl-7337-a890cc\" (UID: \"189f3c86-4927-4f46-8634-b9c94c893452\") " pod="opendatahub/test-curl-7337-a890cc" Oct 30 19:59:00.003665 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:00.003572 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-a890cc" Oct 30 19:59:00.006993 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:00.006973 4292 generic.go:356] "Generic (PLEG): container finished" podID="3845b969-7d5b-48d4-a2d8-dfbe93f91c05" containerID="0470224b914e5efac94b13fd57065c9111b37a70a940942abe7b68b24b4dc94a" exitCode=0 Oct 30 19:59:00.007075 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:00.007051 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f" event={"ID":"3845b969-7d5b-48d4-a2d8-dfbe93f91c05","Type":"ContainerDied","Data":"0470224b914e5efac94b13fd57065c9111b37a70a940942abe7b68b24b4dc94a"} Oct 30 19:59:00.008586 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:00.008564 4292 generic.go:356] "Generic (PLEG): container finished" podID="8efc6802-ecd2-45b1-af43-799ca4e9bcb1" containerID="993f26fffefb36c7206d53e299629013acae48dfa108f4fe48988446cc2c349f" exitCode=0 Oct 30 19:59:00.008714 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:00.008598 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj" event={"ID":"8efc6802-ecd2-45b1-af43-799ca4e9bcb1","Type":"ContainerDied","Data":"993f26fffefb36c7206d53e299629013acae48dfa108f4fe48988446cc2c349f"} Oct 30 19:59:00.008714 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:00.008615 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj" Oct 30 19:59:00.008714 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:00.008650 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj" event={"ID":"8efc6802-ecd2-45b1-af43-799ca4e9bcb1","Type":"ContainerDied","Data":"4ab08da98feb787db61715e1d7a3cbcec7e3634f68550e51ea30064740a078dc"} Oct 30 19:59:00.008714 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:00.008664 4292 scope.go:122] "RemoveContainer" containerID="993f26fffefb36c7206d53e299629013acae48dfa108f4fe48988446cc2c349f" Oct 30 19:59:00.010452 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:00.010432 4292 generic.go:356] "Generic (PLEG): container finished" podID="b6c13eb3-0acd-4a2e-9171-28042161aa6f" containerID="590b8e96b4ad68e9e46ffccfcc9e537d9fb1ad4d9be5ff138cc28a9785d99b86" exitCode=0 Oct 30 19:59:00.010564 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:00.010511 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e67696-6796dcfcf6-zncj4" event={"ID":"b6c13eb3-0acd-4a2e-9171-28042161aa6f","Type":"ContainerDied","Data":"590b8e96b4ad68e9e46ffccfcc9e537d9fb1ad4d9be5ff138cc28a9785d99b86"} Oct 30 19:59:00.010564 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:00.010523 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e67696-6796dcfcf6-zncj4" Oct 30 19:59:00.010564 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:00.010545 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e67696-6796dcfcf6-zncj4" event={"ID":"b6c13eb3-0acd-4a2e-9171-28042161aa6f","Type":"ContainerDied","Data":"215ce296a0ebd4dea1093b134a54ac44b1711111c6cc4d8e38b1ff1a783685ae"} Oct 30 19:59:00.020488 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:00.020410 4292 scope.go:122] "RemoveContainer" containerID="993f26fffefb36c7206d53e299629013acae48dfa108f4fe48988446cc2c349f" Oct 30 19:59:00.021098 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:59:00.020800 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"993f26fffefb36c7206d53e299629013acae48dfa108f4fe48988446cc2c349f\": container with ID starting with 993f26fffefb36c7206d53e299629013acae48dfa108f4fe48988446cc2c349f not found: ID does not exist" containerID="993f26fffefb36c7206d53e299629013acae48dfa108f4fe48988446cc2c349f" Oct 30 19:59:00.021098 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:00.020838 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"993f26fffefb36c7206d53e299629013acae48dfa108f4fe48988446cc2c349f"} err="failed to get container status \"993f26fffefb36c7206d53e299629013acae48dfa108f4fe48988446cc2c349f\": rpc error: code = NotFound desc = could not find container \"993f26fffefb36c7206d53e299629013acae48dfa108f4fe48988446cc2c349f\": container with ID starting with 993f26fffefb36c7206d53e299629013acae48dfa108f4fe48988446cc2c349f not found: ID does not exist" Oct 30 19:59:00.021098 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:00.020973 4292 scope.go:122] "RemoveContainer" containerID="590b8e96b4ad68e9e46ffccfcc9e537d9fb1ad4d9be5ff138cc28a9785d99b86" Oct 30 19:59:00.030676 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:00.030652 4292 scope.go:122] "RemoveContainer" containerID="590b8e96b4ad68e9e46ffccfcc9e537d9fb1ad4d9be5ff138cc28a9785d99b86" Oct 30 19:59:00.030987 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:59:00.030958 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"590b8e96b4ad68e9e46ffccfcc9e537d9fb1ad4d9be5ff138cc28a9785d99b86\": container with ID starting with 590b8e96b4ad68e9e46ffccfcc9e537d9fb1ad4d9be5ff138cc28a9785d99b86 not found: ID does not exist" containerID="590b8e96b4ad68e9e46ffccfcc9e537d9fb1ad4d9be5ff138cc28a9785d99b86" Oct 30 19:59:00.031079 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:00.030998 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"590b8e96b4ad68e9e46ffccfcc9e537d9fb1ad4d9be5ff138cc28a9785d99b86"} err="failed to get container status \"590b8e96b4ad68e9e46ffccfcc9e537d9fb1ad4d9be5ff138cc28a9785d99b86\": rpc error: code = NotFound desc = could not find container \"590b8e96b4ad68e9e46ffccfcc9e537d9fb1ad4d9be5ff138cc28a9785d99b86\": container with ID starting with 590b8e96b4ad68e9e46ffccfcc9e537d9fb1ad4d9be5ff138cc28a9785d99b86 not found: ID does not exist" Oct 30 19:59:00.079397 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:00.077766 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f" Oct 30 19:59:00.084646 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:00.084610 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3845b969-7d5b-48d4-a2d8-dfbe93f91c05-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3845b969-7d5b-48d4-a2d8-dfbe93f91c05-credential-socket\") pod \"3845b969-7d5b-48d4-a2d8-dfbe93f91c05\" (UID: \"3845b969-7d5b-48d4-a2d8-dfbe93f91c05\") " Oct 30 19:59:00.084895 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:00.084872 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3845b969-7d5b-48d4-a2d8-dfbe93f91c05-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3845b969-7d5b-48d4-a2d8-dfbe93f91c05-workload-certs\") pod \"3845b969-7d5b-48d4-a2d8-dfbe93f91c05\" (UID: \"3845b969-7d5b-48d4-a2d8-dfbe93f91c05\") " Oct 30 19:59:00.085134 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:00.085113 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3845b969-7d5b-48d4-a2d8-dfbe93f91c05-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3845b969-7d5b-48d4-a2d8-dfbe93f91c05-istio-podinfo\") pod \"3845b969-7d5b-48d4-a2d8-dfbe93f91c05\" (UID: \"3845b969-7d5b-48d4-a2d8-dfbe93f91c05\") " Oct 30 19:59:00.085319 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:00.085283 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3845b969-7d5b-48d4-a2d8-dfbe93f91c05-istio-token\" (UniqueName: \"kubernetes.io/projected/3845b969-7d5b-48d4-a2d8-dfbe93f91c05-istio-token\") pod \"3845b969-7d5b-48d4-a2d8-dfbe93f91c05\" (UID: \"3845b969-7d5b-48d4-a2d8-dfbe93f91c05\") " Oct 30 19:59:00.085457 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:00.085428 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3845b969-7d5b-48d4-a2d8-dfbe93f91c05-workload-certs" pod "3845b969-7d5b-48d4-a2d8-dfbe93f91c05" (UID: "3845b969-7d5b-48d4-a2d8-dfbe93f91c05"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:59:00.085979 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:00.085960 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3845b969-7d5b-48d4-a2d8-dfbe93f91c05-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3845b969-7d5b-48d4-a2d8-dfbe93f91c05-istio-data\") pod \"3845b969-7d5b-48d4-a2d8-dfbe93f91c05\" (UID: \"3845b969-7d5b-48d4-a2d8-dfbe93f91c05\") " Oct 30 19:59:00.086107 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:00.086093 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3845b969-7d5b-48d4-a2d8-dfbe93f91c05-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3845b969-7d5b-48d4-a2d8-dfbe93f91c05-istio-envoy\") pod \"3845b969-7d5b-48d4-a2d8-dfbe93f91c05\" (UID: \"3845b969-7d5b-48d4-a2d8-dfbe93f91c05\") " Oct 30 19:59:00.086217 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:00.086204 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3845b969-7d5b-48d4-a2d8-dfbe93f91c05-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3845b969-7d5b-48d4-a2d8-dfbe93f91c05-istiod-ca-cert\") pod \"3845b969-7d5b-48d4-a2d8-dfbe93f91c05\" (UID: \"3845b969-7d5b-48d4-a2d8-dfbe93f91c05\") " Oct 30 19:59:00.086328 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:00.086315 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3845b969-7d5b-48d4-a2d8-dfbe93f91c05-kube-api-access-52q4m\" (UniqueName: \"kubernetes.io/projected/3845b969-7d5b-48d4-a2d8-dfbe93f91c05-kube-api-access-52q4m\") pod \"3845b969-7d5b-48d4-a2d8-dfbe93f91c05\" (UID: \"3845b969-7d5b-48d4-a2d8-dfbe93f91c05\") " Oct 30 19:59:00.086458 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:00.086445 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3845b969-7d5b-48d4-a2d8-dfbe93f91c05-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3845b969-7d5b-48d4-a2d8-dfbe93f91c05-workload-socket\") pod \"3845b969-7d5b-48d4-a2d8-dfbe93f91c05\" (UID: \"3845b969-7d5b-48d4-a2d8-dfbe93f91c05\") " Oct 30 19:59:00.086990 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:00.086968 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3845b969-7d5b-48d4-a2d8-dfbe93f91c05-workload-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:00.087657 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:00.087290 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3845b969-7d5b-48d4-a2d8-dfbe93f91c05-istio-data" pod "3845b969-7d5b-48d4-a2d8-dfbe93f91c05" (UID: "3845b969-7d5b-48d4-a2d8-dfbe93f91c05"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:59:00.087657 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:00.087304 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3845b969-7d5b-48d4-a2d8-dfbe93f91c05-workload-socket" pod "3845b969-7d5b-48d4-a2d8-dfbe93f91c05" (UID: "3845b969-7d5b-48d4-a2d8-dfbe93f91c05"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:59:00.087657 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:00.087480 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3845b969-7d5b-48d4-a2d8-dfbe93f91c05-istiod-ca-cert" pod "3845b969-7d5b-48d4-a2d8-dfbe93f91c05" (UID: "3845b969-7d5b-48d4-a2d8-dfbe93f91c05"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:59:00.088627 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:00.088600 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3845b969-7d5b-48d4-a2d8-dfbe93f91c05-credential-socket" pod "3845b969-7d5b-48d4-a2d8-dfbe93f91c05" (UID: "3845b969-7d5b-48d4-a2d8-dfbe93f91c05"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:59:00.089700 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:00.089672 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3845b969-7d5b-48d4-a2d8-dfbe93f91c05-istio-podinfo" pod "3845b969-7d5b-48d4-a2d8-dfbe93f91c05" (UID: "3845b969-7d5b-48d4-a2d8-dfbe93f91c05"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:59:00.089793 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:00.089716 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3845b969-7d5b-48d4-a2d8-dfbe93f91c05-istio-token" pod "3845b969-7d5b-48d4-a2d8-dfbe93f91c05" (UID: "3845b969-7d5b-48d4-a2d8-dfbe93f91c05"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:59:00.092292 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:00.092004 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3845b969-7d5b-48d4-a2d8-dfbe93f91c05-istio-envoy" pod "3845b969-7d5b-48d4-a2d8-dfbe93f91c05" (UID: "3845b969-7d5b-48d4-a2d8-dfbe93f91c05"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:59:00.092664 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:00.092638 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3845b969-7d5b-48d4-a2d8-dfbe93f91c05-kube-api-access-52q4m" pod "3845b969-7d5b-48d4-a2d8-dfbe93f91c05" (UID: "3845b969-7d5b-48d4-a2d8-dfbe93f91c05"). InnerVolumeSpecName "kube-api-access-52q4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:59:00.185622 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:00.185588 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj"] Oct 30 19:59:00.187784 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:00.187757 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3845b969-7d5b-48d4-a2d8-dfbe93f91c05-istio-podinfo\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:00.187784 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:00.187782 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3845b969-7d5b-48d4-a2d8-dfbe93f91c05-istio-token\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:00.187955 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:00.187793 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3845b969-7d5b-48d4-a2d8-dfbe93f91c05-istio-data\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:00.187955 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:00.187802 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3845b969-7d5b-48d4-a2d8-dfbe93f91c05-istio-envoy\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:00.187955 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:00.187811 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3845b969-7d5b-48d4-a2d8-dfbe93f91c05-istiod-ca-cert\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:00.187955 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:00.187819 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52q4m\" (UniqueName: \"kubernetes.io/projected/3845b969-7d5b-48d4-a2d8-dfbe93f91c05-kube-api-access-52q4m\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:00.187955 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:00.187830 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3845b969-7d5b-48d4-a2d8-dfbe93f91c05-workload-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:00.187955 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:00.187838 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3845b969-7d5b-48d4-a2d8-dfbe93f91c05-credential-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:00.189555 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:00.189531 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-z52kj"] Oct 30 19:59:00.302587 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:00.302560 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e67696-6796dcfcf6-zncj4"] Oct 30 19:59:00.304662 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:00.304642 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e67696-6796dcfcf6-zncj4"] Oct 30 19:59:00.407016 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:00.406989 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7337-a890cc"] Oct 30 19:59:00.408091 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:59:00.408063 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod189f3c86_4927_4f46_8634_b9c94c893452.slice/crio-7840a73c0366a9f0c631d4f5021e475b62248498065a79168f024ea144ccfaa9 WatchSource:0}: Error finding container 7840a73c0366a9f0c631d4f5021e475b62248498065a79168f024ea144ccfaa9: Status 404 returned error can't find the container with id 7840a73c0366a9f0c631d4f5021e475b62248498065a79168f024ea144ccfaa9 Oct 30 19:59:00.410016 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:00.409999 4292 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:59:00.582204 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:00.582134 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8efc6802-ecd2-45b1-af43-799ca4e9bcb1" path="/var/lib/kubelet/pods/8efc6802-ecd2-45b1-af43-799ca4e9bcb1/volumes" Oct 30 19:59:00.582607 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:00.582593 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6c13eb3-0acd-4a2e-9171-28042161aa6f" path="/var/lib/kubelet/pods/b6c13eb3-0acd-4a2e-9171-28042161aa6f/volumes" Oct 30 19:59:01.017584 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:01.017551 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7337-a890cc" event={"ID":"189f3c86-4927-4f46-8634-b9c94c893452","Type":"ContainerStarted","Data":"7840a73c0366a9f0c631d4f5021e475b62248498065a79168f024ea144ccfaa9"} Oct 30 19:59:01.018864 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:01.018833 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f" event={"ID":"3845b969-7d5b-48d4-a2d8-dfbe93f91c05","Type":"ContainerDied","Data":"ade2d2b271557a9e48b069568b2b81beb502947e4e950c1afbceace2768a045d"} Oct 30 19:59:01.018990 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:01.018887 4292 scope.go:122] "RemoveContainer" containerID="0470224b914e5efac94b13fd57065c9111b37a70a940942abe7b68b24b4dc94a" Oct 30 19:59:01.018990 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:01.018837 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f" Oct 30 19:59:01.189649 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:01.189617 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f"] Oct 30 19:59:01.239865 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:01.239823 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-44n5f"] Oct 30 19:59:02.584508 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:02.584472 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3845b969-7d5b-48d4-a2d8-dfbe93f91c05" path="/var/lib/kubelet/pods/3845b969-7d5b-48d4-a2d8-dfbe93f91c05/volumes" Oct 30 19:59:04.040413 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:04.040375 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7337-a890cc" event={"ID":"189f3c86-4927-4f46-8634-b9c94c893452","Type":"ContainerStarted","Data":"0e9031a3cf4bdd89be08407ac7d66b867dc915ddc188a04740ce93e083bb478b"} Oct 30 19:59:04.161124 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:04.161073 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7337-a890cc" podStartSLOduration=2.177023755 podStartE2EDuration="5.161060653s" totalImagesPullingTime="2.984036898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:04.160308517 +0000 UTC m=+2047.678623976" watchObservedRunningTime="2026-10-30 19:59:04.161060653 +0000 UTC m=+2047.679376109" Oct 30 19:59:05.728122 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:05.728090 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7337-a890cc"] Oct 30 19:59:05.730714 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:05.730686 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7337-a890cc"] Oct 30 19:59:06.052418 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:06.052319 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7337-a890cc" podUID="189f3c86-4927-4f46-8634-b9c94c893452" containerName="test-curl-7337-a890cc" containerID="cri-o://0e9031a3cf4bdd89be08407ac7d66b867dc915ddc188a04740ce93e083bb478b" gracePeriod=2 Oct 30 19:59:06.104590 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:59:06.104549 4292 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7337-a890cc\" is forbidden: User \"system:node:ip-10-0-131-147.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-147.us-west-2.compute.internal' and this object" podUID="189f3c86-4927-4f46-8634-b9c94c893452" pod="opendatahub/test-curl-7337-a890cc" Oct 30 19:59:06.634196 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:59:06.634161 4292 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7337-a890cc\" is forbidden: User \"system:node:ip-10-0-131-147.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-147.us-west-2.compute.internal' and this object" podUID="189f3c86-4927-4f46-8634-b9c94c893452" pod="opendatahub/test-curl-7337-a890cc" Oct 30 19:59:08.277409 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:08.277387 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-a890cc" Oct 30 19:59:08.378398 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:08.378309 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/189f3c86-4927-4f46-8634-b9c94c893452-kube-api-access-r5wm8\" (UniqueName: \"kubernetes.io/projected/189f3c86-4927-4f46-8634-b9c94c893452-kube-api-access-r5wm8\") pod \"189f3c86-4927-4f46-8634-b9c94c893452\" (UID: \"189f3c86-4927-4f46-8634-b9c94c893452\") " Oct 30 19:59:08.380314 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:08.380286 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/189f3c86-4927-4f46-8634-b9c94c893452-kube-api-access-r5wm8" pod "189f3c86-4927-4f46-8634-b9c94c893452" (UID: "189f3c86-4927-4f46-8634-b9c94c893452"). InnerVolumeSpecName "kube-api-access-r5wm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:59:08.479315 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:08.479287 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5wm8\" (UniqueName: \"kubernetes.io/projected/189f3c86-4927-4f46-8634-b9c94c893452-kube-api-access-r5wm8\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:08.581769 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:08.581741 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="189f3c86-4927-4f46-8634-b9c94c893452" path="/var/lib/kubelet/pods/189f3c86-4927-4f46-8634-b9c94c893452/volumes" Oct 30 19:59:09.070951 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:09.070915 4292 generic.go:356] "Generic (PLEG): container finished" podID="189f3c86-4927-4f46-8634-b9c94c893452" containerID="0e9031a3cf4bdd89be08407ac7d66b867dc915ddc188a04740ce93e083bb478b" exitCode=137 Oct 30 19:59:09.071153 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:09.070966 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-a890cc" Oct 30 19:59:09.071153 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:09.070976 4292 scope.go:122] "RemoveContainer" containerID="0e9031a3cf4bdd89be08407ac7d66b867dc915ddc188a04740ce93e083bb478b" Oct 30 19:59:09.079839 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:09.079814 4292 scope.go:122] "RemoveContainer" containerID="0e9031a3cf4bdd89be08407ac7d66b867dc915ddc188a04740ce93e083bb478b" Oct 30 19:59:09.080115 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:59:09.080095 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e9031a3cf4bdd89be08407ac7d66b867dc915ddc188a04740ce93e083bb478b\": container with ID starting with 0e9031a3cf4bdd89be08407ac7d66b867dc915ddc188a04740ce93e083bb478b not found: ID does not exist" containerID="0e9031a3cf4bdd89be08407ac7d66b867dc915ddc188a04740ce93e083bb478b" Oct 30 19:59:09.080196 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:09.080122 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e9031a3cf4bdd89be08407ac7d66b867dc915ddc188a04740ce93e083bb478b"} err="failed to get container status \"0e9031a3cf4bdd89be08407ac7d66b867dc915ddc188a04740ce93e083bb478b\": rpc error: code = NotFound desc = could not find container \"0e9031a3cf4bdd89be08407ac7d66b867dc915ddc188a04740ce93e083bb478b\": container with ID starting with 0e9031a3cf4bdd89be08407ac7d66b867dc915ddc188a04740ce93e083bb478b not found: ID does not exist" Oct 30 19:59:11.410613 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:11.410581 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6"] Oct 30 19:59:11.411030 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:11.411016 4292 cpu_manager.go:414] "containerMap: removing container" podUID="189f3c86-4927-4f46-8634-b9c94c893452" containerName="test-curl-7337-a890cc" Oct 30 19:59:11.411094 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:11.411031 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="189f3c86-4927-4f46-8634-b9c94c893452" containerName="test-curl-7337-a890cc" Oct 30 19:59:11.411094 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:11.411042 4292 cpu_manager.go:414] "containerMap: removing container" podUID="b6c13eb3-0acd-4a2e-9171-28042161aa6f" containerName="maas-api" Oct 30 19:59:11.411094 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:11.411048 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6c13eb3-0acd-4a2e-9171-28042161aa6f" containerName="maas-api" Oct 30 19:59:11.411094 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:11.411070 4292 cpu_manager.go:414] "containerMap: removing container" podUID="3845b969-7d5b-48d4-a2d8-dfbe93f91c05" containerName="istio-proxy" Oct 30 19:59:11.411094 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:11.411075 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3845b969-7d5b-48d4-a2d8-dfbe93f91c05" containerName="istio-proxy" Oct 30 19:59:11.411094 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:11.411086 4292 cpu_manager.go:414] "containerMap: removing container" podUID="02e3cbb1-bde4-4a15-b543-db2b4b820f99" containerName="istio-proxy" Oct 30 19:59:11.411094 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:11.411092 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02e3cbb1-bde4-4a15-b543-db2b4b820f99" containerName="istio-proxy" Oct 30 19:59:11.411303 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:11.411160 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="3845b969-7d5b-48d4-a2d8-dfbe93f91c05" containerName="istio-proxy" Oct 30 19:59:11.411303 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:11.411172 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="189f3c86-4927-4f46-8634-b9c94c893452" containerName="test-curl-7337-a890cc" Oct 30 19:59:11.411303 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:11.411178 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6c13eb3-0acd-4a2e-9171-28042161aa6f" containerName="maas-api" Oct 30 19:59:11.411303 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:11.411185 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="02e3cbb1-bde4-4a15-b543-db2b4b820f99" containerName="istio-proxy" Oct 30 19:59:11.415498 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:11.415477 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6" Oct 30 19:59:11.513265 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:11.513230 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6"] Oct 30 19:59:11.520181 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:11.520155 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-ffe14b-openshift-default-dockercfg-kpx9f\"" Oct 30 19:59:11.520319 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:11.520227 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/554fd8be-cebc-4035-840e-e377169383e5-istio-podinfo\") pod \"e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6\" (UID: \"554fd8be-cebc-4035-840e-e377169383e5\") " pod="openshift-ingress/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6" Oct 30 19:59:11.520319 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:11.520273 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/554fd8be-cebc-4035-840e-e377169383e5-istio-envoy\") pod \"e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6\" (UID: \"554fd8be-cebc-4035-840e-e377169383e5\") " pod="openshift-ingress/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6" Oct 30 19:59:11.520434 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:11.520401 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/554fd8be-cebc-4035-840e-e377169383e5-workload-socket\") pod \"e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6\" (UID: \"554fd8be-cebc-4035-840e-e377169383e5\") " pod="openshift-ingress/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6" Oct 30 19:59:11.520472 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:11.520437 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/554fd8be-cebc-4035-840e-e377169383e5-istio-data\") pod \"e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6\" (UID: \"554fd8be-cebc-4035-840e-e377169383e5\") " pod="openshift-ingress/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6" Oct 30 19:59:11.520472 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:11.520461 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kckt\" (UniqueName: \"kubernetes.io/projected/554fd8be-cebc-4035-840e-e377169383e5-kube-api-access-6kckt\") pod \"e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6\" (UID: \"554fd8be-cebc-4035-840e-e377169383e5\") " pod="openshift-ingress/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6" Oct 30 19:59:11.520636 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:11.520619 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/554fd8be-cebc-4035-840e-e377169383e5-istio-token\") pod \"e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6\" (UID: \"554fd8be-cebc-4035-840e-e377169383e5\") " pod="openshift-ingress/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6" Oct 30 19:59:11.520779 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:11.520756 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/554fd8be-cebc-4035-840e-e377169383e5-workload-certs\") pod \"e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6\" (UID: \"554fd8be-cebc-4035-840e-e377169383e5\") " pod="openshift-ingress/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6" Oct 30 19:59:11.520841 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:11.520824 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/554fd8be-cebc-4035-840e-e377169383e5-credential-socket\") pod \"e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6\" (UID: \"554fd8be-cebc-4035-840e-e377169383e5\") " pod="openshift-ingress/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6" Oct 30 19:59:11.520880 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:11.520855 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/554fd8be-cebc-4035-840e-e377169383e5-istiod-ca-cert\") pod \"e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6\" (UID: \"554fd8be-cebc-4035-840e-e377169383e5\") " pod="openshift-ingress/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6" Oct 30 19:59:11.621741 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:11.621694 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/554fd8be-cebc-4035-840e-e377169383e5-workload-certs\") pod \"e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6\" (UID: \"554fd8be-cebc-4035-840e-e377169383e5\") " pod="openshift-ingress/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6" Oct 30 19:59:11.621937 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:11.621753 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/554fd8be-cebc-4035-840e-e377169383e5-credential-socket\") pod \"e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6\" (UID: \"554fd8be-cebc-4035-840e-e377169383e5\") " pod="openshift-ingress/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6" Oct 30 19:59:11.621937 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:11.621779 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/554fd8be-cebc-4035-840e-e377169383e5-istiod-ca-cert\") pod \"e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6\" (UID: \"554fd8be-cebc-4035-840e-e377169383e5\") " pod="openshift-ingress/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6" Oct 30 19:59:11.621937 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:11.621888 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/554fd8be-cebc-4035-840e-e377169383e5-istio-podinfo\") pod \"e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6\" (UID: \"554fd8be-cebc-4035-840e-e377169383e5\") " pod="openshift-ingress/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6" Oct 30 19:59:11.621937 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:11.621918 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/554fd8be-cebc-4035-840e-e377169383e5-istio-envoy\") pod \"e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6\" (UID: \"554fd8be-cebc-4035-840e-e377169383e5\") " pod="openshift-ingress/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6" Oct 30 19:59:11.622170 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:11.621953 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/554fd8be-cebc-4035-840e-e377169383e5-workload-socket\") pod \"e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6\" (UID: \"554fd8be-cebc-4035-840e-e377169383e5\") " pod="openshift-ingress/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6" Oct 30 19:59:11.622170 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:11.622132 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/554fd8be-cebc-4035-840e-e377169383e5-istio-data\") pod \"e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6\" (UID: \"554fd8be-cebc-4035-840e-e377169383e5\") " pod="openshift-ingress/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6" Oct 30 19:59:11.622281 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:11.622176 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kckt\" (UniqueName: \"kubernetes.io/projected/554fd8be-cebc-4035-840e-e377169383e5-kube-api-access-6kckt\") pod \"e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6\" (UID: \"554fd8be-cebc-4035-840e-e377169383e5\") " pod="openshift-ingress/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6" Oct 30 19:59:11.622281 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:11.622182 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/554fd8be-cebc-4035-840e-e377169383e5-workload-certs\") pod \"e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6\" (UID: \"554fd8be-cebc-4035-840e-e377169383e5\") " pod="openshift-ingress/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6" Oct 30 19:59:11.622281 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:11.622213 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/554fd8be-cebc-4035-840e-e377169383e5-credential-socket\") pod \"e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6\" (UID: \"554fd8be-cebc-4035-840e-e377169383e5\") " pod="openshift-ingress/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6" Oct 30 19:59:11.622448 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:11.622274 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/554fd8be-cebc-4035-840e-e377169383e5-istio-token\") pod \"e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6\" (UID: \"554fd8be-cebc-4035-840e-e377169383e5\") " pod="openshift-ingress/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6" Oct 30 19:59:11.622506 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:11.622475 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/554fd8be-cebc-4035-840e-e377169383e5-istio-data\") pod \"e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6\" (UID: \"554fd8be-cebc-4035-840e-e377169383e5\") " pod="openshift-ingress/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6" Oct 30 19:59:11.622637 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:11.622612 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/554fd8be-cebc-4035-840e-e377169383e5-istiod-ca-cert\") pod \"e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6\" (UID: \"554fd8be-cebc-4035-840e-e377169383e5\") " pod="openshift-ingress/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6" Oct 30 19:59:11.622637 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:11.622624 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/554fd8be-cebc-4035-840e-e377169383e5-workload-socket\") pod \"e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6\" (UID: \"554fd8be-cebc-4035-840e-e377169383e5\") " pod="openshift-ingress/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6" Oct 30 19:59:11.625455 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:11.625432 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/554fd8be-cebc-4035-840e-e377169383e5-istio-envoy\") pod \"e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6\" (UID: \"554fd8be-cebc-4035-840e-e377169383e5\") " pod="openshift-ingress/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6" Oct 30 19:59:11.625747 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:11.625695 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/554fd8be-cebc-4035-840e-e377169383e5-istio-podinfo\") pod \"e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6\" (UID: \"554fd8be-cebc-4035-840e-e377169383e5\") " pod="openshift-ingress/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6" Oct 30 19:59:11.689840 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:11.689763 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kckt\" (UniqueName: \"kubernetes.io/projected/554fd8be-cebc-4035-840e-e377169383e5-kube-api-access-6kckt\") pod \"e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6\" (UID: \"554fd8be-cebc-4035-840e-e377169383e5\") " pod="openshift-ingress/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6" Oct 30 19:59:11.698449 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:11.698415 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/554fd8be-cebc-4035-840e-e377169383e5-istio-token\") pod \"e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6\" (UID: \"554fd8be-cebc-4035-840e-e377169383e5\") " pod="openshift-ingress/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6" Oct 30 19:59:11.728577 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:11.728549 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6" Oct 30 19:59:12.160870 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:12.160845 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6"] Oct 30 19:59:12.163339 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:59:12.163296 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod554fd8be_cebc_4035_840e_e377169383e5.slice/crio-c470489af1bedccf2346d2dc5c4b6a2df788ab28016e5f7560b68e2f73823736 WatchSource:0}: Error finding container c470489af1bedccf2346d2dc5c4b6a2df788ab28016e5f7560b68e2f73823736: Status 404 returned error can't find the container with id c470489af1bedccf2346d2dc5c4b6a2df788ab28016e5f7560b68e2f73823736 Oct 30 19:59:12.165825 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:12.165792 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:59:12.165942 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:12.165858 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:59:12.165942 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:12.165890 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:59:13.107684 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:13.107651 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6" event={"ID":"554fd8be-cebc-4035-840e-e377169383e5","Type":"ContainerStarted","Data":"4d37c3a61cf245489e3daf66c689410e3d7155c5ed1263cb202415cdd7d6d768"} Oct 30 19:59:13.108207 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:13.107718 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6" event={"ID":"554fd8be-cebc-4035-840e-e377169383e5","Type":"ContainerStarted","Data":"c470489af1bedccf2346d2dc5c4b6a2df788ab28016e5f7560b68e2f73823736"} Oct 30 19:59:13.219954 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:59:13.219909 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:59:13.256889 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:13.256840 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6" podStartSLOduration=2.256826297 podStartE2EDuration="2.256826297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:13.254784431 +0000 UTC m=+2056.773099974" watchObservedRunningTime="2026-10-30 19:59:13.256826297 +0000 UTC m=+2056.775141754" Oct 30 19:59:13.729044 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:13.729013 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6" Oct 30 19:59:13.734549 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:13.734527 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6" Oct 30 19:59:14.112711 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:14.112673 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6" Oct 30 19:59:14.113746 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:14.113728 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6" Oct 30 19:59:27.463185 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:27.463143 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ffe14b-76d78847cb-gq7tv"] Oct 30 19:59:27.468500 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:27.468476 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ffe14b-76d78847cb-gq7tv" Oct 30 19:59:27.493484 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:27.493454 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ffe14b-76d78847cb-gq7tv"] Oct 30 19:59:27.514018 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:27.513983 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3759813-8a90-4e72-b94d-86634ee99202-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ffe14b-76d78847cb-gq7tv\" (UID: \"c3759813-8a90-4e72-b94d-86634ee99202\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ffe14b-76d78847cb-gq7tv" Oct 30 19:59:27.514171 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:27.514058 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhplf\" (UniqueName: \"kubernetes.io/projected/c3759813-8a90-4e72-b94d-86634ee99202-kube-api-access-vhplf\") pod \"payload-processing-e2e-ipp-a-ffe14b-76d78847cb-gq7tv\" (UID: \"c3759813-8a90-4e72-b94d-86634ee99202\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ffe14b-76d78847cb-gq7tv" Oct 30 19:59:27.540437 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:27.540410 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-ffe14b\"" Oct 30 19:59:27.540582 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:27.540468 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-ffe14b-dockercfg-fjrlr\"" Oct 30 19:59:27.614884 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:27.614848 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3759813-8a90-4e72-b94d-86634ee99202-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ffe14b-76d78847cb-gq7tv\" (UID: \"c3759813-8a90-4e72-b94d-86634ee99202\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ffe14b-76d78847cb-gq7tv" Oct 30 19:59:27.615087 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:27.614905 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhplf\" (UniqueName: \"kubernetes.io/projected/c3759813-8a90-4e72-b94d-86634ee99202-kube-api-access-vhplf\") pod \"payload-processing-e2e-ipp-a-ffe14b-76d78847cb-gq7tv\" (UID: \"c3759813-8a90-4e72-b94d-86634ee99202\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ffe14b-76d78847cb-gq7tv" Oct 30 19:59:27.615636 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:27.615612 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3759813-8a90-4e72-b94d-86634ee99202-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ffe14b-76d78847cb-gq7tv\" (UID: \"c3759813-8a90-4e72-b94d-86634ee99202\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ffe14b-76d78847cb-gq7tv" Oct 30 19:59:27.677391 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:27.677342 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhplf\" (UniqueName: \"kubernetes.io/projected/c3759813-8a90-4e72-b94d-86634ee99202-kube-api-access-vhplf\") pod \"payload-processing-e2e-ipp-a-ffe14b-76d78847cb-gq7tv\" (UID: \"c3759813-8a90-4e72-b94d-86634ee99202\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ffe14b-76d78847cb-gq7tv" Oct 30 19:59:27.785524 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:27.785493 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ffe14b-76d78847cb-gq7tv" Oct 30 19:59:28.160777 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:28.160745 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ffe14b-76d78847cb-gq7tv"] Oct 30 19:59:28.161462 ip-10-0-131-147 kubenswrapper[4292]: W1030 19:59:28.161436 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3759813_8a90_4e72_b94d_86634ee99202.slice/crio-cde5076a0bc630def4d0bc4e3bf258eb940c74c5b70a8018126db7bc850b8ce8 WatchSource:0}: Error finding container cde5076a0bc630def4d0bc4e3bf258eb940c74c5b70a8018126db7bc850b8ce8: Status 404 returned error can't find the container with id cde5076a0bc630def4d0bc4e3bf258eb940c74c5b70a8018126db7bc850b8ce8 Oct 30 19:59:28.193099 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:28.193072 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ffe14b-76d78847cb-gq7tv" event={"ID":"c3759813-8a90-4e72-b94d-86634ee99202","Type":"ContainerStarted","Data":"cde5076a0bc630def4d0bc4e3bf258eb940c74c5b70a8018126db7bc850b8ce8"} Oct 30 19:59:29.241592 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:29.241554 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ffe14b-76d78847cb-gq7tv" event={"ID":"c3759813-8a90-4e72-b94d-86634ee99202","Type":"ContainerStarted","Data":"35052d8644c74695b12a245878c333e6a4646c02e8fa90c160fc93ca60b052b9"} Oct 30 19:59:29.242450 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:29.242428 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ffe14b-76d78847cb-gq7tv" Oct 30 19:59:29.361469 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:29.361413 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-ffe14b-76d78847cb-gq7tv" podStartSLOduration=2.361398959 podStartE2EDuration="2.361398959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:29.360031695 +0000 UTC m=+2072.878347172" watchObservedRunningTime="2026-10-30 19:59:29.361398959 +0000 UTC m=+2072.879714416" Oct 30 19:59:41.254720 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:41.254690 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ffe14b-76d78847cb-gq7tv" Oct 30 19:59:43.221819 ip-10-0-131-147 kubenswrapper[4292]: E1030 19:59:43.221783 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 19:59:45.473321 ip-10-0-131-147 kubenswrapper[4292]: I1030 19:59:45.473283 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-111f9c64/e2e-lifecycle-model-111f9c64-kserve-5ffc645545-lm7mr"] Oct 30 20:00:00.174283 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:00.174238 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889840-dpnvp"] Oct 30 20:00:00.179204 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:00.179185 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-585635-29889840-zpbrt"] Oct 30 20:00:00.179373 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:00.179314 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889840-dpnvp" Oct 30 20:00:00.182702 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:00.182682 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d80c7d-29889840-q5c97"] Oct 30 20:00:00.182823 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:00.182784 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-585635-29889840-zpbrt" Oct 30 20:00:00.185966 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:00.185946 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d80c7d-29889840-q5c97" Oct 30 20:00:00.224529 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:00.224504 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889840-6x7xp"] Oct 30 20:00:00.227853 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:00.227836 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889840-6x7xp" Oct 30 20:00:00.233135 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:00.233114 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\"" Oct 30 20:00:00.267475 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:00.267447 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-585635-29889840-zpbrt"] Oct 30 20:00:00.276251 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:00.276213 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889840-dpnvp"] Oct 30 20:00:00.278430 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:00.278404 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d80c7d-29889840-q5c97"] Oct 30 20:00:00.313383 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:00.311620 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88rmx\" (UniqueName: \"kubernetes.io/projected/875f38d8-03c7-452d-bfd5-e2fc26a738bd-kube-api-access-88rmx\") pod \"maas-api-key-cleanup-e2e-ipp-a-d80c7d-29889840-q5c97\" (UID: \"875f38d8-03c7-452d-bfd5-e2fc26a738bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d80c7d-29889840-q5c97" Oct 30 20:00:00.313383 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:00.311675 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx7bv\" (UniqueName: \"kubernetes.io/projected/5c58d708-c81a-4f9f-95d2-dc7590a46f8b-kube-api-access-mx7bv\") pod \"maas-api-key-cleanup-e2e-worker-w2-b0a183-29889840-dpnvp\" (UID: \"5c58d708-c81a-4f9f-95d2-dc7590a46f8b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889840-dpnvp" Oct 30 20:00:00.313383 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:00.313178 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xfmp\" (UniqueName: \"kubernetes.io/projected/0363aa9f-fcfb-4ddc-b950-23441ea7f137-kube-api-access-5xfmp\") pod \"maas-api-key-cleanup-e2e-api-b-585635-29889840-zpbrt\" (UID: \"0363aa9f-fcfb-4ddc-b950-23441ea7f137\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-585635-29889840-zpbrt" Oct 30 20:00:00.313383 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:00.313222 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/875f38d8-03c7-452d-bfd5-e2fc26a738bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-d80c7d-29889840-q5c97\" (UID: \"875f38d8-03c7-452d-bfd5-e2fc26a738bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d80c7d-29889840-q5c97" Oct 30 20:00:00.313383 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:00.313339 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c58d708-c81a-4f9f-95d2-dc7590a46f8b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-b0a183-29889840-dpnvp\" (UID: \"5c58d708-c81a-4f9f-95d2-dc7590a46f8b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889840-dpnvp" Oct 30 20:00:00.313383 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:00.313387 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0363aa9f-fcfb-4ddc-b950-23441ea7f137-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-585635-29889840-zpbrt\" (UID: \"0363aa9f-fcfb-4ddc-b950-23441ea7f137\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-585635-29889840-zpbrt" Oct 30 20:00:00.313827 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:00.313429 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9xbh\" (UniqueName: \"kubernetes.io/projected/b8c60c3f-f8e7-4980-8823-b573b514da41-kube-api-access-h9xbh\") pod \"maas-api-key-cleanup-e2e-worker-w4-4ec609-29889840-6x7xp\" (UID: \"b8c60c3f-f8e7-4980-8823-b573b514da41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889840-6x7xp" Oct 30 20:00:00.313827 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:00.313489 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8c60c3f-f8e7-4980-8823-b573b514da41-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-4ec609-29889840-6x7xp\" (UID: \"b8c60c3f-f8e7-4980-8823-b573b514da41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889840-6x7xp" Oct 30 20:00:00.314504 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:00.314476 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889840-6x7xp"] Oct 30 20:00:00.415113 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:00.415026 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xfmp\" (UniqueName: \"kubernetes.io/projected/0363aa9f-fcfb-4ddc-b950-23441ea7f137-kube-api-access-5xfmp\") pod \"maas-api-key-cleanup-e2e-api-b-585635-29889840-zpbrt\" (UID: \"0363aa9f-fcfb-4ddc-b950-23441ea7f137\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-585635-29889840-zpbrt" Oct 30 20:00:00.454588 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:00.454498 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/875f38d8-03c7-452d-bfd5-e2fc26a738bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-d80c7d-29889840-q5c97\" (UID: \"875f38d8-03c7-452d-bfd5-e2fc26a738bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d80c7d-29889840-q5c97" Oct 30 20:00:00.454761 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:00.454648 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c58d708-c81a-4f9f-95d2-dc7590a46f8b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-b0a183-29889840-dpnvp\" (UID: \"5c58d708-c81a-4f9f-95d2-dc7590a46f8b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889840-dpnvp" Oct 30 20:00:00.454761 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:00.454683 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0363aa9f-fcfb-4ddc-b950-23441ea7f137-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-585635-29889840-zpbrt\" (UID: \"0363aa9f-fcfb-4ddc-b950-23441ea7f137\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-585635-29889840-zpbrt" Oct 30 20:00:00.454761 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:00.454724 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9xbh\" (UniqueName: \"kubernetes.io/projected/b8c60c3f-f8e7-4980-8823-b573b514da41-kube-api-access-h9xbh\") pod \"maas-api-key-cleanup-e2e-worker-w4-4ec609-29889840-6x7xp\" (UID: \"b8c60c3f-f8e7-4980-8823-b573b514da41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889840-6x7xp" Oct 30 20:00:00.454946 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:00.454783 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8c60c3f-f8e7-4980-8823-b573b514da41-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-4ec609-29889840-6x7xp\" (UID: \"b8c60c3f-f8e7-4980-8823-b573b514da41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889840-6x7xp" Oct 30 20:00:00.454946 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:00.454844 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88rmx\" (UniqueName: \"kubernetes.io/projected/875f38d8-03c7-452d-bfd5-e2fc26a738bd-kube-api-access-88rmx\") pod \"maas-api-key-cleanup-e2e-ipp-a-d80c7d-29889840-q5c97\" (UID: \"875f38d8-03c7-452d-bfd5-e2fc26a738bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d80c7d-29889840-q5c97" Oct 30 20:00:00.454946 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:00.454879 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx7bv\" (UniqueName: \"kubernetes.io/projected/5c58d708-c81a-4f9f-95d2-dc7590a46f8b-kube-api-access-mx7bv\") pod \"maas-api-key-cleanup-e2e-worker-w2-b0a183-29889840-dpnvp\" (UID: \"5c58d708-c81a-4f9f-95d2-dc7590a46f8b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889840-dpnvp" Oct 30 20:00:00.456053 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:00.456023 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/875f38d8-03c7-452d-bfd5-e2fc26a738bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-d80c7d-29889840-q5c97\" (UID: \"875f38d8-03c7-452d-bfd5-e2fc26a738bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d80c7d-29889840-q5c97" Oct 30 20:00:00.457078 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:00.456397 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c58d708-c81a-4f9f-95d2-dc7590a46f8b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-b0a183-29889840-dpnvp\" (UID: \"5c58d708-c81a-4f9f-95d2-dc7590a46f8b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889840-dpnvp" Oct 30 20:00:00.457078 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:00.456894 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8c60c3f-f8e7-4980-8823-b573b514da41-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-4ec609-29889840-6x7xp\" (UID: \"b8c60c3f-f8e7-4980-8823-b573b514da41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889840-6x7xp" Oct 30 20:00:00.457425 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:00.457398 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0363aa9f-fcfb-4ddc-b950-23441ea7f137-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-585635-29889840-zpbrt\" (UID: \"0363aa9f-fcfb-4ddc-b950-23441ea7f137\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-585635-29889840-zpbrt" Oct 30 20:00:00.477886 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:00.477859 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xfmp\" (UniqueName: \"kubernetes.io/projected/0363aa9f-fcfb-4ddc-b950-23441ea7f137-kube-api-access-5xfmp\") pod \"maas-api-key-cleanup-e2e-api-b-585635-29889840-zpbrt\" (UID: \"0363aa9f-fcfb-4ddc-b950-23441ea7f137\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-585635-29889840-zpbrt" Oct 30 20:00:00.493545 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:00.493498 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-111f9c64/e2e-lifecycle-model-111f9c64-kserve-5ffc645545-lm7mr" podUID="d25c22f7-bb65-4300-9332-872a7be76774" containerName="main" containerID="cri-o://662cc166ec13b7e039b4da8cf9668432e400448d37dbf370249db0fcf41cd6ba" gracePeriod=45 Oct 30 20:00:00.501047 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:00.501024 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-585635-29889840-zpbrt" Oct 30 20:00:00.518026 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:00.517995 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx7bv\" (UniqueName: \"kubernetes.io/projected/5c58d708-c81a-4f9f-95d2-dc7590a46f8b-kube-api-access-mx7bv\") pod \"maas-api-key-cleanup-e2e-worker-w2-b0a183-29889840-dpnvp\" (UID: \"5c58d708-c81a-4f9f-95d2-dc7590a46f8b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889840-dpnvp" Oct 30 20:00:00.518457 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:00.518433 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9xbh\" (UniqueName: \"kubernetes.io/projected/b8c60c3f-f8e7-4980-8823-b573b514da41-kube-api-access-h9xbh\") pod \"maas-api-key-cleanup-e2e-worker-w4-4ec609-29889840-6x7xp\" (UID: \"b8c60c3f-f8e7-4980-8823-b573b514da41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889840-6x7xp" Oct 30 20:00:00.519726 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:00.519579 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88rmx\" (UniqueName: \"kubernetes.io/projected/875f38d8-03c7-452d-bfd5-e2fc26a738bd-kube-api-access-88rmx\") pod \"maas-api-key-cleanup-e2e-ipp-a-d80c7d-29889840-q5c97\" (UID: \"875f38d8-03c7-452d-bfd5-e2fc26a738bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d80c7d-29889840-q5c97" Oct 30 20:00:00.540324 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:00.540301 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889840-6x7xp" Oct 30 20:00:00.759431 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:00.759403 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-111f9c64/e2e-lifecycle-model-111f9c64-kserve-5ffc645545-lm7mr" Oct 30 20:00:00.794633 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:00.794600 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889840-dpnvp" Oct 30 20:00:00.807228 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:00.807206 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d80c7d-29889840-q5c97" Oct 30 20:00:00.864106 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:00.864075 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d25c22f7-bb65-4300-9332-872a7be76774-tls-certs\" (UniqueName: \"kubernetes.io/secret/d25c22f7-bb65-4300-9332-872a7be76774-tls-certs\") pod \"d25c22f7-bb65-4300-9332-872a7be76774\" (UID: \"d25c22f7-bb65-4300-9332-872a7be76774\") " Oct 30 20:00:00.864172 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:00.864138 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d25c22f7-bb65-4300-9332-872a7be76774-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d25c22f7-bb65-4300-9332-872a7be76774-tmp-dir\") pod \"d25c22f7-bb65-4300-9332-872a7be76774\" (UID: \"d25c22f7-bb65-4300-9332-872a7be76774\") " Oct 30 20:00:00.864218 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:00.864203 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d25c22f7-bb65-4300-9332-872a7be76774-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d25c22f7-bb65-4300-9332-872a7be76774-dshm\") pod \"d25c22f7-bb65-4300-9332-872a7be76774\" (UID: \"d25c22f7-bb65-4300-9332-872a7be76774\") " Oct 30 20:00:00.864340 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:00.864324 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d25c22f7-bb65-4300-9332-872a7be76774-kube-api-access-76tzf\" (UniqueName: \"kubernetes.io/projected/d25c22f7-bb65-4300-9332-872a7be76774-kube-api-access-76tzf\") pod \"d25c22f7-bb65-4300-9332-872a7be76774\" (UID: \"d25c22f7-bb65-4300-9332-872a7be76774\") " Oct 30 20:00:00.864440 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:00.864398 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d25c22f7-bb65-4300-9332-872a7be76774-home\" (UniqueName: \"kubernetes.io/empty-dir/d25c22f7-bb65-4300-9332-872a7be76774-home\") pod \"d25c22f7-bb65-4300-9332-872a7be76774\" (UID: \"d25c22f7-bb65-4300-9332-872a7be76774\") " Oct 30 20:00:00.864495 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:00.864438 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d25c22f7-bb65-4300-9332-872a7be76774-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d25c22f7-bb65-4300-9332-872a7be76774-model-cache\") pod \"d25c22f7-bb65-4300-9332-872a7be76774\" (UID: \"d25c22f7-bb65-4300-9332-872a7be76774\") " Oct 30 20:00:00.865217 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:00.865187 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d25c22f7-bb65-4300-9332-872a7be76774-model-cache" pod "d25c22f7-bb65-4300-9332-872a7be76774" (UID: "d25c22f7-bb65-4300-9332-872a7be76774"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:00.872660 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:00.866032 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d25c22f7-bb65-4300-9332-872a7be76774-tmp-dir" pod "d25c22f7-bb65-4300-9332-872a7be76774" (UID: "d25c22f7-bb65-4300-9332-872a7be76774"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:00.876633 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:00.874860 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d25c22f7-bb65-4300-9332-872a7be76774-home" pod "d25c22f7-bb65-4300-9332-872a7be76774" (UID: "d25c22f7-bb65-4300-9332-872a7be76774"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:00.882988 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:00.882596 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-585635-29889840-zpbrt"] Oct 30 20:00:00.895393 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:00.891756 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d25c22f7-bb65-4300-9332-872a7be76774-dshm" pod "d25c22f7-bb65-4300-9332-872a7be76774" (UID: "d25c22f7-bb65-4300-9332-872a7be76774"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:00.895393 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:00:00.891839 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0363aa9f_fcfb_4ddc_b950_23441ea7f137.slice/crio-70cdb7f4e54dff01fa01dd194f2afd0817f5fd72b3e824610fee37cfafe217f7 WatchSource:0}: Error finding container 70cdb7f4e54dff01fa01dd194f2afd0817f5fd72b3e824610fee37cfafe217f7: Status 404 returned error can't find the container with id 70cdb7f4e54dff01fa01dd194f2afd0817f5fd72b3e824610fee37cfafe217f7 Oct 30 20:00:00.896651 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:00.896628 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d25c22f7-bb65-4300-9332-872a7be76774-tls-certs" pod "d25c22f7-bb65-4300-9332-872a7be76774" (UID: "d25c22f7-bb65-4300-9332-872a7be76774"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:00:00.898671 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:00.898644 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d25c22f7-bb65-4300-9332-872a7be76774-kube-api-access-76tzf" pod "d25c22f7-bb65-4300-9332-872a7be76774" (UID: "d25c22f7-bb65-4300-9332-872a7be76774"). InnerVolumeSpecName "kube-api-access-76tzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:00.956055 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:00.955866 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889840-6x7xp"] Oct 30 20:00:00.956879 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:00:00.956840 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8c60c3f_f8e7_4980_8823_b573b514da41.slice/crio-923cd5147318c243f645438670ae6a0124e1db649d9bcab59cce89d3a7640ca9 WatchSource:0}: Error finding container 923cd5147318c243f645438670ae6a0124e1db649d9bcab59cce89d3a7640ca9: Status 404 returned error can't find the container with id 923cd5147318c243f645438670ae6a0124e1db649d9bcab59cce89d3a7640ca9 Oct 30 20:00:00.965800 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:00.965764 4292 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d25c22f7-bb65-4300-9332-872a7be76774-tls-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:00.965898 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:00.965800 4292 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d25c22f7-bb65-4300-9332-872a7be76774-tmp-dir\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:00.965898 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:00.965816 4292 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d25c22f7-bb65-4300-9332-872a7be76774-dshm\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:00.965898 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:00.965832 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76tzf\" (UniqueName: \"kubernetes.io/projected/d25c22f7-bb65-4300-9332-872a7be76774-kube-api-access-76tzf\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:00.965898 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:00.965848 4292 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d25c22f7-bb65-4300-9332-872a7be76774-home\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:00.965898 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:00.965862 4292 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d25c22f7-bb65-4300-9332-872a7be76774-model-cache\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:01.192684 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:01.192643 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889840-dpnvp"] Oct 30 20:00:01.193645 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:00:01.193616 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c58d708_c81a_4f9f_95d2_dc7590a46f8b.slice/crio-00e64b685bd17c6ab70f54134c6622daf0894ae60dbc31af91d6b4fa56540e69 WatchSource:0}: Error finding container 00e64b685bd17c6ab70f54134c6622daf0894ae60dbc31af91d6b4fa56540e69: Status 404 returned error can't find the container with id 00e64b685bd17c6ab70f54134c6622daf0894ae60dbc31af91d6b4fa56540e69 Oct 30 20:00:01.195095 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:01.195074 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d80c7d-29889840-q5c97"] Oct 30 20:00:01.197938 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:00:01.197911 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod875f38d8_03c7_452d_bfd5_e2fc26a738bd.slice/crio-5f2328b0fbc1324c78d649fb7ca4feafe33ea01be4e8ebf35fec31a874604a7e WatchSource:0}: Error finding container 5f2328b0fbc1324c78d649fb7ca4feafe33ea01be4e8ebf35fec31a874604a7e: Status 404 returned error can't find the container with id 5f2328b0fbc1324c78d649fb7ca4feafe33ea01be4e8ebf35fec31a874604a7e Oct 30 20:00:01.457403 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:01.457336 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d80c7d-29889840-q5c97" event={"ID":"875f38d8-03c7-452d-bfd5-e2fc26a738bd","Type":"ContainerStarted","Data":"5f2328b0fbc1324c78d649fb7ca4feafe33ea01be4e8ebf35fec31a874604a7e"} Oct 30 20:00:01.458621 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:01.458595 4292 generic.go:356] "Generic (PLEG): container finished" podID="d25c22f7-bb65-4300-9332-872a7be76774" containerID="662cc166ec13b7e039b4da8cf9668432e400448d37dbf370249db0fcf41cd6ba" exitCode=0 Oct 30 20:00:01.458757 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:01.458650 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-111f9c64/e2e-lifecycle-model-111f9c64-kserve-5ffc645545-lm7mr" Oct 30 20:00:01.458757 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:01.458671 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-111f9c64/e2e-lifecycle-model-111f9c64-kserve-5ffc645545-lm7mr" event={"ID":"d25c22f7-bb65-4300-9332-872a7be76774","Type":"ContainerDied","Data":"662cc166ec13b7e039b4da8cf9668432e400448d37dbf370249db0fcf41cd6ba"} Oct 30 20:00:01.458757 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:01.458713 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-111f9c64/e2e-lifecycle-model-111f9c64-kserve-5ffc645545-lm7mr" event={"ID":"d25c22f7-bb65-4300-9332-872a7be76774","Type":"ContainerDied","Data":"fecdaa8b03b679c52d15a758e729dc42fff8ee886ba6eaa619de0bdd95ce34e3"} Oct 30 20:00:01.458757 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:01.458737 4292 scope.go:122] "RemoveContainer" containerID="662cc166ec13b7e039b4da8cf9668432e400448d37dbf370249db0fcf41cd6ba" Oct 30 20:00:01.460420 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:01.460387 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-585635-29889840-zpbrt" event={"ID":"0363aa9f-fcfb-4ddc-b950-23441ea7f137","Type":"ContainerStarted","Data":"1ddcffc1341292fafa85434fe49f72b21bb12104ada52ea3356cd116b744ad7c"} Oct 30 20:00:01.460510 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:01.460448 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-585635-29889840-zpbrt" event={"ID":"0363aa9f-fcfb-4ddc-b950-23441ea7f137","Type":"ContainerStarted","Data":"70cdb7f4e54dff01fa01dd194f2afd0817f5fd72b3e824610fee37cfafe217f7"} Oct 30 20:00:01.461968 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:01.461948 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889840-dpnvp" event={"ID":"5c58d708-c81a-4f9f-95d2-dc7590a46f8b","Type":"ContainerStarted","Data":"00e64b685bd17c6ab70f54134c6622daf0894ae60dbc31af91d6b4fa56540e69"} Oct 30 20:00:01.463940 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:01.463765 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889840-6x7xp" event={"ID":"b8c60c3f-f8e7-4980-8823-b573b514da41","Type":"ContainerStarted","Data":"838ed2bb82fab6610fba4d79b14ba8db44442f8c5198e00d5cb01d12db402234"} Oct 30 20:00:01.463940 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:01.463809 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889840-6x7xp" event={"ID":"b8c60c3f-f8e7-4980-8823-b573b514da41","Type":"ContainerStarted","Data":"923cd5147318c243f645438670ae6a0124e1db649d9bcab59cce89d3a7640ca9"} Oct 30 20:00:01.468853 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:01.468832 4292 scope.go:122] "RemoveContainer" containerID="662cc166ec13b7e039b4da8cf9668432e400448d37dbf370249db0fcf41cd6ba" Oct 30 20:00:01.469106 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:00:01.469083 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"662cc166ec13b7e039b4da8cf9668432e400448d37dbf370249db0fcf41cd6ba\": container with ID starting with 662cc166ec13b7e039b4da8cf9668432e400448d37dbf370249db0fcf41cd6ba not found: ID does not exist" containerID="662cc166ec13b7e039b4da8cf9668432e400448d37dbf370249db0fcf41cd6ba" Oct 30 20:00:01.469163 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:01.469114 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"662cc166ec13b7e039b4da8cf9668432e400448d37dbf370249db0fcf41cd6ba"} err="failed to get container status \"662cc166ec13b7e039b4da8cf9668432e400448d37dbf370249db0fcf41cd6ba\": rpc error: code = NotFound desc = could not find container \"662cc166ec13b7e039b4da8cf9668432e400448d37dbf370249db0fcf41cd6ba\": container with ID starting with 662cc166ec13b7e039b4da8cf9668432e400448d37dbf370249db0fcf41cd6ba not found: ID does not exist" Oct 30 20:00:01.632140 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:01.632047 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-585635-29889840-zpbrt" podStartSLOduration=1.632032742 podStartE2EDuration="1.632032742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:01.629739564 +0000 UTC m=+2105.148055020" watchObservedRunningTime="2026-10-30 20:00:01.632032742 +0000 UTC m=+2105.150348199" Oct 30 20:00:01.752899 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:01.752841 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889840-6x7xp" podStartSLOduration=1.752826352 podStartE2EDuration="1.752826352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:01.74980946 +0000 UTC m=+2105.268124944" watchObservedRunningTime="2026-10-30 20:00:01.752826352 +0000 UTC m=+2105.271141809" Oct 30 20:00:01.868170 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:01.868142 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-111f9c64/e2e-lifecycle-model-111f9c64-kserve-5ffc645545-lm7mr"] Oct 30 20:00:01.875115 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:01.875089 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-111f9c64/e2e-lifecycle-model-111f9c64-kserve-5ffc645545-lm7mr"] Oct 30 20:00:02.472540 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:02.472460 4292 generic.go:356] "Generic (PLEG): container finished" podID="b8c60c3f-f8e7-4980-8823-b573b514da41" containerID="838ed2bb82fab6610fba4d79b14ba8db44442f8c5198e00d5cb01d12db402234" exitCode=60 Oct 30 20:00:02.472973 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:02.472542 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889840-6x7xp" event={"ID":"b8c60c3f-f8e7-4980-8823-b573b514da41","Type":"ContainerDied","Data":"838ed2bb82fab6610fba4d79b14ba8db44442f8c5198e00d5cb01d12db402234"} Oct 30 20:00:02.472973 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:02.472781 4292 scope.go:122] "RemoveContainer" containerID="838ed2bb82fab6610fba4d79b14ba8db44442f8c5198e00d5cb01d12db402234" Oct 30 20:00:02.474383 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:02.474216 4292 generic.go:356] "Generic (PLEG): container finished" podID="875f38d8-03c7-452d-bfd5-e2fc26a738bd" containerID="ebbbd042b7a9ef7fbee29265bd7b89ac06ce55db8b7914913135c6eefb44e425" exitCode=60 Oct 30 20:00:02.474383 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:02.474258 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d80c7d-29889840-q5c97" event={"ID":"875f38d8-03c7-452d-bfd5-e2fc26a738bd","Type":"ContainerDied","Data":"ebbbd042b7a9ef7fbee29265bd7b89ac06ce55db8b7914913135c6eefb44e425"} Oct 30 20:00:02.474577 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:02.474560 4292 scope.go:122] "RemoveContainer" containerID="ebbbd042b7a9ef7fbee29265bd7b89ac06ce55db8b7914913135c6eefb44e425" Oct 30 20:00:02.477014 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:02.476994 4292 generic.go:356] "Generic (PLEG): container finished" podID="0363aa9f-fcfb-4ddc-b950-23441ea7f137" containerID="1ddcffc1341292fafa85434fe49f72b21bb12104ada52ea3356cd116b744ad7c" exitCode=60 Oct 30 20:00:02.477107 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:02.477088 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-585635-29889840-zpbrt" event={"ID":"0363aa9f-fcfb-4ddc-b950-23441ea7f137","Type":"ContainerDied","Data":"1ddcffc1341292fafa85434fe49f72b21bb12104ada52ea3356cd116b744ad7c"} Oct 30 20:00:02.477321 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:02.477298 4292 scope.go:122] "RemoveContainer" containerID="1ddcffc1341292fafa85434fe49f72b21bb12104ada52ea3356cd116b744ad7c" Oct 30 20:00:02.478774 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:02.478753 4292 generic.go:356] "Generic (PLEG): container finished" podID="5c58d708-c81a-4f9f-95d2-dc7590a46f8b" containerID="e81edb17ad2036312369e61d34c3481202cce99256f88dac1f74055c8c10e8b8" exitCode=60 Oct 30 20:00:02.478870 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:02.478798 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889840-dpnvp" event={"ID":"5c58d708-c81a-4f9f-95d2-dc7590a46f8b","Type":"ContainerDied","Data":"e81edb17ad2036312369e61d34c3481202cce99256f88dac1f74055c8c10e8b8"} Oct 30 20:00:02.479012 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:02.478997 4292 scope.go:122] "RemoveContainer" containerID="e81edb17ad2036312369e61d34c3481202cce99256f88dac1f74055c8c10e8b8" Oct 30 20:00:02.582102 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:02.582073 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d25c22f7-bb65-4300-9332-872a7be76774" path="/var/lib/kubelet/pods/d25c22f7-bb65-4300-9332-872a7be76774/volumes" Oct 30 20:00:03.486929 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:03.486895 4292 generic.go:356] "Generic (PLEG): container finished" podID="b8c60c3f-f8e7-4980-8823-b573b514da41" containerID="f93c8ae7b05c73702ed4ad2e098190ed2a5dafe4eac1d67ca45a3e28d56c831f" exitCode=60 Oct 30 20:00:03.487335 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:03.486974 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889840-6x7xp" event={"ID":"b8c60c3f-f8e7-4980-8823-b573b514da41","Type":"ContainerDied","Data":"f93c8ae7b05c73702ed4ad2e098190ed2a5dafe4eac1d67ca45a3e28d56c831f"} Oct 30 20:00:03.487335 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:03.487044 4292 scope.go:122] "RemoveContainer" containerID="838ed2bb82fab6610fba4d79b14ba8db44442f8c5198e00d5cb01d12db402234" Oct 30 20:00:03.487335 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:03.487316 4292 scope.go:122] "RemoveContainer" containerID="f93c8ae7b05c73702ed4ad2e098190ed2a5dafe4eac1d67ca45a3e28d56c831f" Oct 30 20:00:03.487679 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:00:03.487660 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-4ec609-29889840-6x7xp_odh-ai-gateway-infra(b8c60c3f-f8e7-4980-8823-b573b514da41)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889840-6x7xp" podUID="b8c60c3f-f8e7-4980-8823-b573b514da41" Oct 30 20:00:03.488991 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:03.488965 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d80c7d-29889840-q5c97" event={"ID":"875f38d8-03c7-452d-bfd5-e2fc26a738bd","Type":"ContainerStarted","Data":"3f8241c0a51f6a37c228b029510eb25529e48ef0030ac26a85a1aa081f9ec3eb"} Oct 30 20:00:03.490725 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:03.490703 4292 generic.go:356] "Generic (PLEG): container finished" podID="0363aa9f-fcfb-4ddc-b950-23441ea7f137" containerID="8f58b84351d93820e65d9bc4c6fed3421175c8a540b1a45f5b15b34013d5d048" exitCode=60 Oct 30 20:00:03.490838 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:03.490784 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-585635-29889840-zpbrt" event={"ID":"0363aa9f-fcfb-4ddc-b950-23441ea7f137","Type":"ContainerDied","Data":"8f58b84351d93820e65d9bc4c6fed3421175c8a540b1a45f5b15b34013d5d048"} Oct 30 20:00:03.491019 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:03.491004 4292 scope.go:122] "RemoveContainer" containerID="8f58b84351d93820e65d9bc4c6fed3421175c8a540b1a45f5b15b34013d5d048" Oct 30 20:00:03.491281 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:00:03.491257 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-585635-29889840-zpbrt_odh-ai-gateway-infra(0363aa9f-fcfb-4ddc-b950-23441ea7f137)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-585635-29889840-zpbrt" podUID="0363aa9f-fcfb-4ddc-b950-23441ea7f137" Oct 30 20:00:03.492566 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:03.492545 4292 generic.go:356] "Generic (PLEG): container finished" podID="5c58d708-c81a-4f9f-95d2-dc7590a46f8b" containerID="3364857e6983ae6bc29601df839593a14a63784147af668e22d7a2b52c7e1f44" exitCode=60 Oct 30 20:00:03.492668 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:03.492573 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889840-dpnvp" event={"ID":"5c58d708-c81a-4f9f-95d2-dc7590a46f8b","Type":"ContainerDied","Data":"3364857e6983ae6bc29601df839593a14a63784147af668e22d7a2b52c7e1f44"} Oct 30 20:00:03.492857 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:03.492839 4292 scope.go:122] "RemoveContainer" containerID="3364857e6983ae6bc29601df839593a14a63784147af668e22d7a2b52c7e1f44" Oct 30 20:00:03.493086 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:00:03.493067 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-b0a183-29889840-dpnvp_odh-ai-gateway-infra(5c58d708-c81a-4f9f-95d2-dc7590a46f8b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889840-dpnvp" podUID="5c58d708-c81a-4f9f-95d2-dc7590a46f8b" Oct 30 20:00:03.529685 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:03.529663 4292 scope.go:122] "RemoveContainer" containerID="1ddcffc1341292fafa85434fe49f72b21bb12104ada52ea3356cd116b744ad7c" Oct 30 20:00:03.541806 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:03.541790 4292 scope.go:122] "RemoveContainer" containerID="e81edb17ad2036312369e61d34c3481202cce99256f88dac1f74055c8c10e8b8" Oct 30 20:00:03.962227 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:03.962172 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d80c7d-29889840-q5c97" podStartSLOduration=3.962156576 podStartE2EDuration="3.962156576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:03.95971638 +0000 UTC m=+2107.478031838" watchObservedRunningTime="2026-10-30 20:00:03.962156576 +0000 UTC m=+2107.480472032" Oct 30 20:00:04.499922 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:04.499883 4292 generic.go:356] "Generic (PLEG): container finished" podID="875f38d8-03c7-452d-bfd5-e2fc26a738bd" containerID="3f8241c0a51f6a37c228b029510eb25529e48ef0030ac26a85a1aa081f9ec3eb" exitCode=60 Oct 30 20:00:04.500454 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:04.499968 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d80c7d-29889840-q5c97" event={"ID":"875f38d8-03c7-452d-bfd5-e2fc26a738bd","Type":"ContainerDied","Data":"3f8241c0a51f6a37c228b029510eb25529e48ef0030ac26a85a1aa081f9ec3eb"} Oct 30 20:00:04.500454 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:04.500063 4292 scope.go:122] "RemoveContainer" containerID="ebbbd042b7a9ef7fbee29265bd7b89ac06ce55db8b7914913135c6eefb44e425" Oct 30 20:00:04.500454 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:04.500270 4292 scope.go:122] "RemoveContainer" containerID="3f8241c0a51f6a37c228b029510eb25529e48ef0030ac26a85a1aa081f9ec3eb" Oct 30 20:00:04.500663 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:00:04.500641 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-d80c7d-29889840-q5c97_odh-ai-gateway-infra(875f38d8-03c7-452d-bfd5-e2fc26a738bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d80c7d-29889840-q5c97" podUID="875f38d8-03c7-452d-bfd5-e2fc26a738bd" Oct 30 20:00:04.501981 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:04.501963 4292 scope.go:122] "RemoveContainer" containerID="8f58b84351d93820e65d9bc4c6fed3421175c8a540b1a45f5b15b34013d5d048" Oct 30 20:00:04.502252 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:00:04.502228 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-585635-29889840-zpbrt_odh-ai-gateway-infra(0363aa9f-fcfb-4ddc-b950-23441ea7f137)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-585635-29889840-zpbrt" podUID="0363aa9f-fcfb-4ddc-b950-23441ea7f137" Oct 30 20:00:04.503570 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:04.503551 4292 scope.go:122] "RemoveContainer" containerID="3364857e6983ae6bc29601df839593a14a63784147af668e22d7a2b52c7e1f44" Oct 30 20:00:04.503832 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:00:04.503798 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-b0a183-29889840-dpnvp_odh-ai-gateway-infra(5c58d708-c81a-4f9f-95d2-dc7590a46f8b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889840-dpnvp" podUID="5c58d708-c81a-4f9f-95d2-dc7590a46f8b" Oct 30 20:00:04.505138 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:04.505122 4292 scope.go:122] "RemoveContainer" containerID="f93c8ae7b05c73702ed4ad2e098190ed2a5dafe4eac1d67ca45a3e28d56c831f" Oct 30 20:00:04.505328 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:00:04.505312 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-4ec609-29889840-6x7xp_odh-ai-gateway-infra(b8c60c3f-f8e7-4980-8823-b573b514da41)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889840-6x7xp" podUID="b8c60c3f-f8e7-4980-8823-b573b514da41" Oct 30 20:00:05.513680 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:05.513654 4292 scope.go:122] "RemoveContainer" containerID="3f8241c0a51f6a37c228b029510eb25529e48ef0030ac26a85a1aa081f9ec3eb" Oct 30 20:00:05.514062 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:00:05.513933 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-d80c7d-29889840-q5c97_odh-ai-gateway-infra(875f38d8-03c7-452d-bfd5-e2fc26a738bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d80c7d-29889840-q5c97" podUID="875f38d8-03c7-452d-bfd5-e2fc26a738bd" Oct 30 20:00:08.329985 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:08.329946 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr"] Oct 30 20:00:08.330535 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:08.330518 4292 cpu_manager.go:414] "containerMap: removing container" podUID="d25c22f7-bb65-4300-9332-872a7be76774" containerName="main" Oct 30 20:00:08.330583 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:08.330536 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d25c22f7-bb65-4300-9332-872a7be76774" containerName="main" Oct 30 20:00:08.330617 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:08.330610 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="d25c22f7-bb65-4300-9332-872a7be76774" containerName="main" Oct 30 20:00:08.335214 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:08.335190 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr" Oct 30 20:00:08.395587 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:08.395555 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-8b0edc-openshift-default-dockercfg-94zlg\"" Oct 30 20:00:08.450713 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:08.450672 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr"] Oct 30 20:00:08.470986 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:08.470944 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe-istiod-ca-cert\") pod \"e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr\" (UID: \"cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe\") " pod="openshift-ingress/e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr" Oct 30 20:00:08.471303 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:08.471268 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe-credential-socket\") pod \"e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr\" (UID: \"cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe\") " pod="openshift-ingress/e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr" Oct 30 20:00:08.471600 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:08.471571 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe-workload-socket\") pod \"e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr\" (UID: \"cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe\") " pod="openshift-ingress/e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr" Oct 30 20:00:08.471727 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:08.471611 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe-workload-certs\") pod \"e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr\" (UID: \"cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe\") " pod="openshift-ingress/e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr" Oct 30 20:00:08.471727 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:08.471715 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe-istio-envoy\") pod \"e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr\" (UID: \"cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe\") " pod="openshift-ingress/e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr" Oct 30 20:00:08.471842 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:08.471805 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe-istio-data\") pod \"e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr\" (UID: \"cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe\") " pod="openshift-ingress/e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr" Oct 30 20:00:08.471842 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:08.471831 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plj6d\" (UniqueName: \"kubernetes.io/projected/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe-kube-api-access-plj6d\") pod \"e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr\" (UID: \"cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe\") " pod="openshift-ingress/e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr" Oct 30 20:00:08.471945 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:08.471867 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe-istio-token\") pod \"e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr\" (UID: \"cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe\") " pod="openshift-ingress/e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr" Oct 30 20:00:08.471945 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:08.471923 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe-istio-podinfo\") pod \"e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr\" (UID: \"cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe\") " pod="openshift-ingress/e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr" Oct 30 20:00:08.573308 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:08.573271 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe-istiod-ca-cert\") pod \"e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr\" (UID: \"cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe\") " pod="openshift-ingress/e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr" Oct 30 20:00:08.573526 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:08.573320 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe-credential-socket\") pod \"e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr\" (UID: \"cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe\") " pod="openshift-ingress/e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr" Oct 30 20:00:08.573526 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:08.573393 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe-workload-socket\") pod \"e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr\" (UID: \"cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe\") " pod="openshift-ingress/e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr" Oct 30 20:00:08.573526 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:08.573418 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe-workload-certs\") pod \"e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr\" (UID: \"cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe\") " pod="openshift-ingress/e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr" Oct 30 20:00:08.573526 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:08.573516 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe-istio-envoy\") pod \"e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr\" (UID: \"cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe\") " pod="openshift-ingress/e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr" Oct 30 20:00:08.573754 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:08.573604 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe-istio-data\") pod \"e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr\" (UID: \"cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe\") " pod="openshift-ingress/e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr" Oct 30 20:00:08.573754 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:08.573635 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plj6d\" (UniqueName: \"kubernetes.io/projected/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe-kube-api-access-plj6d\") pod \"e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr\" (UID: \"cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe\") " pod="openshift-ingress/e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr" Oct 30 20:00:08.573754 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:08.573674 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe-istio-token\") pod \"e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr\" (UID: \"cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe\") " pod="openshift-ingress/e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr" Oct 30 20:00:08.573754 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:08.573732 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe-istio-podinfo\") pod \"e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr\" (UID: \"cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe\") " pod="openshift-ingress/e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr" Oct 30 20:00:08.573942 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:08.573774 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe-credential-socket\") pod \"e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr\" (UID: \"cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe\") " pod="openshift-ingress/e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr" Oct 30 20:00:08.574378 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:08.574320 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe-istio-data\") pod \"e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr\" (UID: \"cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe\") " pod="openshift-ingress/e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr" Oct 30 20:00:08.574529 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:08.574510 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe-istiod-ca-cert\") pod \"e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr\" (UID: \"cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe\") " pod="openshift-ingress/e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr" Oct 30 20:00:08.574632 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:08.574578 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe-workload-socket\") pod \"e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr\" (UID: \"cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe\") " pod="openshift-ingress/e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr" Oct 30 20:00:08.574841 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:08.574819 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe-workload-certs\") pod \"e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr\" (UID: \"cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe\") " pod="openshift-ingress/e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr" Oct 30 20:00:08.577018 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:08.576993 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe-istio-podinfo\") pod \"e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr\" (UID: \"cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe\") " pod="openshift-ingress/e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr" Oct 30 20:00:08.577866 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:08.577846 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe-istio-envoy\") pod \"e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr\" (UID: \"cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe\") " pod="openshift-ingress/e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr" Oct 30 20:00:08.649423 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:08.646179 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe-istio-token\") pod \"e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr\" (UID: \"cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe\") " pod="openshift-ingress/e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr" Oct 30 20:00:08.649640 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:08.649474 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plj6d\" (UniqueName: \"kubernetes.io/projected/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe-kube-api-access-plj6d\") pod \"e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr\" (UID: \"cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe\") " pod="openshift-ingress/e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr" Oct 30 20:00:08.949086 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:08.949000 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr" Oct 30 20:00:09.392176 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:09.392154 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr"] Oct 30 20:00:09.392955 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:00:09.392929 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd3ad97f_a09f_4932_b5f5_b4a0a8640ebe.slice/crio-9b60291c83f0c65cd4893573ba4ce9209e1e3fd1888c951fa6a2605f69a7760d WatchSource:0}: Error finding container 9b60291c83f0c65cd4893573ba4ce9209e1e3fd1888c951fa6a2605f69a7760d: Status 404 returned error can't find the container with id 9b60291c83f0c65cd4893573ba4ce9209e1e3fd1888c951fa6a2605f69a7760d Oct 30 20:00:09.395633 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:09.395596 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:00:09.395706 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:09.395686 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:00:09.395744 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:09.395731 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:00:09.544388 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:09.544343 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr" event={"ID":"cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe","Type":"ContainerStarted","Data":"6b0c868cf678f8da347762ae591ea92b7ead5baa887c9be3d21cb5dc98c83c61"} Oct 30 20:00:09.544566 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:09.544416 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr" event={"ID":"cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe","Type":"ContainerStarted","Data":"9b60291c83f0c65cd4893573ba4ce9209e1e3fd1888c951fa6a2605f69a7760d"} Oct 30 20:00:09.669950 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:09.669852 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr" podStartSLOduration=1.669839687 podStartE2EDuration="1.669839687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:09.668651144 +0000 UTC m=+2113.186966614" watchObservedRunningTime="2026-10-30 20:00:09.669839687 +0000 UTC m=+2113.188155143" Oct 30 20:00:09.949599 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:09.949518 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr" Oct 30 20:00:10.968177 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:10.968149 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr" Oct 30 20:00:11.556594 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:11.556561 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr" Oct 30 20:00:11.557580 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:11.557557 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr" Oct 30 20:00:13.221035 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:00:13.220990 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:00:14.576410 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:14.576381 4292 scope.go:122] "RemoveContainer" containerID="8f58b84351d93820e65d9bc4c6fed3421175c8a540b1a45f5b15b34013d5d048" Oct 30 20:00:15.581243 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:15.581213 4292 generic.go:356] "Generic (PLEG): container finished" podID="0363aa9f-fcfb-4ddc-b950-23441ea7f137" containerID="398109e00e0b8a35385de0df4d59ff8490451a1f8e7449bf05111643295a270f" exitCode=60 Oct 30 20:00:15.581858 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:15.581285 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-585635-29889840-zpbrt" event={"ID":"0363aa9f-fcfb-4ddc-b950-23441ea7f137","Type":"ContainerDied","Data":"398109e00e0b8a35385de0df4d59ff8490451a1f8e7449bf05111643295a270f"} Oct 30 20:00:15.581858 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:15.581360 4292 scope.go:122] "RemoveContainer" containerID="8f58b84351d93820e65d9bc4c6fed3421175c8a540b1a45f5b15b34013d5d048" Oct 30 20:00:15.581858 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:15.581703 4292 scope.go:122] "RemoveContainer" containerID="398109e00e0b8a35385de0df4d59ff8490451a1f8e7449bf05111643295a270f" Oct 30 20:00:15.582016 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:00:15.581968 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-585635-29889840-zpbrt_odh-ai-gateway-infra(0363aa9f-fcfb-4ddc-b950-23441ea7f137)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-585635-29889840-zpbrt" podUID="0363aa9f-fcfb-4ddc-b950-23441ea7f137" Oct 30 20:00:16.714328 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:16.714295 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-585635-29889840-zpbrt"] Oct 30 20:00:16.895230 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:16.895198 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-585635-29889840-zpbrt" Oct 30 20:00:16.988475 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:16.988397 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0363aa9f-fcfb-4ddc-b950-23441ea7f137-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0363aa9f-fcfb-4ddc-b950-23441ea7f137-maas-api-service-ca\") pod \"0363aa9f-fcfb-4ddc-b950-23441ea7f137\" (UID: \"0363aa9f-fcfb-4ddc-b950-23441ea7f137\") " Oct 30 20:00:16.988623 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:16.988527 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0363aa9f-fcfb-4ddc-b950-23441ea7f137-kube-api-access-5xfmp\" (UniqueName: \"kubernetes.io/projected/0363aa9f-fcfb-4ddc-b950-23441ea7f137-kube-api-access-5xfmp\") pod \"0363aa9f-fcfb-4ddc-b950-23441ea7f137\" (UID: \"0363aa9f-fcfb-4ddc-b950-23441ea7f137\") " Oct 30 20:00:16.988810 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:16.988787 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0363aa9f-fcfb-4ddc-b950-23441ea7f137-maas-api-service-ca" pod "0363aa9f-fcfb-4ddc-b950-23441ea7f137" (UID: "0363aa9f-fcfb-4ddc-b950-23441ea7f137"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:16.988965 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:16.988940 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0363aa9f-fcfb-4ddc-b950-23441ea7f137-maas-api-service-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:16.991093 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:16.991067 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0363aa9f-fcfb-4ddc-b950-23441ea7f137-kube-api-access-5xfmp" pod "0363aa9f-fcfb-4ddc-b950-23441ea7f137" (UID: "0363aa9f-fcfb-4ddc-b950-23441ea7f137"). InnerVolumeSpecName "kube-api-access-5xfmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:17.090258 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:17.090224 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xfmp\" (UniqueName: \"kubernetes.io/projected/0363aa9f-fcfb-4ddc-b950-23441ea7f137-kube-api-access-5xfmp\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:17.577028 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:17.576999 4292 scope.go:122] "RemoveContainer" containerID="3364857e6983ae6bc29601df839593a14a63784147af668e22d7a2b52c7e1f44" Oct 30 20:00:17.577770 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:17.577511 4292 scope.go:122] "RemoveContainer" containerID="f93c8ae7b05c73702ed4ad2e098190ed2a5dafe4eac1d67ca45a3e28d56c831f" Oct 30 20:00:17.577770 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:17.577650 4292 scope.go:122] "RemoveContainer" containerID="3f8241c0a51f6a37c228b029510eb25529e48ef0030ac26a85a1aa081f9ec3eb" Oct 30 20:00:17.596824 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:17.596797 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-585635-29889840-zpbrt" Oct 30 20:00:17.596963 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:17.596868 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-585635-29889840-zpbrt" event={"ID":"0363aa9f-fcfb-4ddc-b950-23441ea7f137","Type":"ContainerDied","Data":"70cdb7f4e54dff01fa01dd194f2afd0817f5fd72b3e824610fee37cfafe217f7"} Oct 30 20:00:17.596963 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:17.596928 4292 scope.go:122] "RemoveContainer" containerID="398109e00e0b8a35385de0df4d59ff8490451a1f8e7449bf05111643295a270f" Oct 30 20:00:17.798362 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:17.798329 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-585635-29889840-zpbrt"] Oct 30 20:00:17.803504 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:17.803476 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-585635-29889840-zpbrt"] Oct 30 20:00:18.597295 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:18.597266 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0363aa9f-fcfb-4ddc-b950-23441ea7f137" path="/var/lib/kubelet/pods/0363aa9f-fcfb-4ddc-b950-23441ea7f137/volumes" Oct 30 20:00:18.608849 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:18.608712 4292 generic.go:356] "Generic (PLEG): container finished" podID="875f38d8-03c7-452d-bfd5-e2fc26a738bd" containerID="1cedffe903b82e20716abe0492a74e893da6b86df3858a670caeb6a64e3e9019" exitCode=60 Oct 30 20:00:18.608849 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:18.608781 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d80c7d-29889840-q5c97" event={"ID":"875f38d8-03c7-452d-bfd5-e2fc26a738bd","Type":"ContainerDied","Data":"1cedffe903b82e20716abe0492a74e893da6b86df3858a670caeb6a64e3e9019"} Oct 30 20:00:18.608849 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:18.608853 4292 scope.go:122] "RemoveContainer" containerID="3f8241c0a51f6a37c228b029510eb25529e48ef0030ac26a85a1aa081f9ec3eb" Oct 30 20:00:18.609522 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:18.609298 4292 scope.go:122] "RemoveContainer" containerID="1cedffe903b82e20716abe0492a74e893da6b86df3858a670caeb6a64e3e9019" Oct 30 20:00:18.609704 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:00:18.609678 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-d80c7d-29889840-q5c97_odh-ai-gateway-infra(875f38d8-03c7-452d-bfd5-e2fc26a738bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d80c7d-29889840-q5c97" podUID="875f38d8-03c7-452d-bfd5-e2fc26a738bd" Oct 30 20:00:18.625917 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:18.622785 4292 generic.go:356] "Generic (PLEG): container finished" podID="5c58d708-c81a-4f9f-95d2-dc7590a46f8b" containerID="4a74a3b448f16fb70a00d538dc673ba09369742931962ab19d7891976e49cafa" exitCode=60 Oct 30 20:00:18.625917 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:18.622899 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889840-dpnvp" event={"ID":"5c58d708-c81a-4f9f-95d2-dc7590a46f8b","Type":"ContainerDied","Data":"4a74a3b448f16fb70a00d538dc673ba09369742931962ab19d7891976e49cafa"} Oct 30 20:00:18.625917 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:18.623262 4292 scope.go:122] "RemoveContainer" containerID="4a74a3b448f16fb70a00d538dc673ba09369742931962ab19d7891976e49cafa" Oct 30 20:00:18.625917 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:00:18.623608 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-b0a183-29889840-dpnvp_odh-ai-gateway-infra(5c58d708-c81a-4f9f-95d2-dc7590a46f8b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889840-dpnvp" podUID="5c58d708-c81a-4f9f-95d2-dc7590a46f8b" Oct 30 20:00:18.630203 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:18.629528 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889840-6x7xp" event={"ID":"b8c60c3f-f8e7-4980-8823-b573b514da41","Type":"ContainerStarted","Data":"964c1b00052909db92392d74432d0f65f24f71d41839185f5eed054b6cb3ac99"} Oct 30 20:00:18.636816 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:18.636795 4292 scope.go:122] "RemoveContainer" containerID="3364857e6983ae6bc29601df839593a14a63784147af668e22d7a2b52c7e1f44" Oct 30 20:00:19.665837 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:19.665796 4292 generic.go:356] "Generic (PLEG): container finished" podID="b8c60c3f-f8e7-4980-8823-b573b514da41" containerID="964c1b00052909db92392d74432d0f65f24f71d41839185f5eed054b6cb3ac99" exitCode=60 Oct 30 20:00:19.666334 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:19.665993 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889840-6x7xp" event={"ID":"b8c60c3f-f8e7-4980-8823-b573b514da41","Type":"ContainerDied","Data":"964c1b00052909db92392d74432d0f65f24f71d41839185f5eed054b6cb3ac99"} Oct 30 20:00:19.666334 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:19.666164 4292 scope.go:122] "RemoveContainer" containerID="f93c8ae7b05c73702ed4ad2e098190ed2a5dafe4eac1d67ca45a3e28d56c831f" Oct 30 20:00:19.667781 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:19.667754 4292 scope.go:122] "RemoveContainer" containerID="964c1b00052909db92392d74432d0f65f24f71d41839185f5eed054b6cb3ac99" Oct 30 20:00:19.668066 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:00:19.668041 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-4ec609-29889840-6x7xp_odh-ai-gateway-infra(b8c60c3f-f8e7-4980-8823-b573b514da41)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889840-6x7xp" podUID="b8c60c3f-f8e7-4980-8823-b573b514da41" Oct 30 20:00:19.827726 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:19.802559 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d80c7d-29889840-q5c97"] Oct 30 20:00:19.893454 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:19.893090 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889840-6x7xp"] Oct 30 20:00:20.000382 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:20.000268 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889840-dpnvp"] Oct 30 20:00:20.039457 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:20.038423 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d80c7d-29889840-q5c97" Oct 30 20:00:20.082205 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:20.082171 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/875f38d8-03c7-452d-bfd5-e2fc26a738bd-kube-api-access-88rmx\" (UniqueName: \"kubernetes.io/projected/875f38d8-03c7-452d-bfd5-e2fc26a738bd-kube-api-access-88rmx\") pod \"875f38d8-03c7-452d-bfd5-e2fc26a738bd\" (UID: \"875f38d8-03c7-452d-bfd5-e2fc26a738bd\") " Oct 30 20:00:20.082388 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:20.082326 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/875f38d8-03c7-452d-bfd5-e2fc26a738bd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/875f38d8-03c7-452d-bfd5-e2fc26a738bd-maas-api-service-ca\") pod \"875f38d8-03c7-452d-bfd5-e2fc26a738bd\" (UID: \"875f38d8-03c7-452d-bfd5-e2fc26a738bd\") " Oct 30 20:00:20.082975 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:20.082931 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/875f38d8-03c7-452d-bfd5-e2fc26a738bd-maas-api-service-ca" pod "875f38d8-03c7-452d-bfd5-e2fc26a738bd" (UID: "875f38d8-03c7-452d-bfd5-e2fc26a738bd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:20.085302 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:20.085273 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/875f38d8-03c7-452d-bfd5-e2fc26a738bd-kube-api-access-88rmx" pod "875f38d8-03c7-452d-bfd5-e2fc26a738bd" (UID: "875f38d8-03c7-452d-bfd5-e2fc26a738bd"). InnerVolumeSpecName "kube-api-access-88rmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:20.167435 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:20.167410 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889840-dpnvp" Oct 30 20:00:20.183983 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:20.183942 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/875f38d8-03c7-452d-bfd5-e2fc26a738bd-maas-api-service-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:20.183983 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:20.183979 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88rmx\" (UniqueName: \"kubernetes.io/projected/875f38d8-03c7-452d-bfd5-e2fc26a738bd-kube-api-access-88rmx\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:20.285545 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:20.284719 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c58d708-c81a-4f9f-95d2-dc7590a46f8b-kube-api-access-mx7bv\" (UniqueName: \"kubernetes.io/projected/5c58d708-c81a-4f9f-95d2-dc7590a46f8b-kube-api-access-mx7bv\") pod \"5c58d708-c81a-4f9f-95d2-dc7590a46f8b\" (UID: \"5c58d708-c81a-4f9f-95d2-dc7590a46f8b\") " Oct 30 20:00:20.285545 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:20.284819 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c58d708-c81a-4f9f-95d2-dc7590a46f8b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c58d708-c81a-4f9f-95d2-dc7590a46f8b-maas-api-service-ca\") pod \"5c58d708-c81a-4f9f-95d2-dc7590a46f8b\" (UID: \"5c58d708-c81a-4f9f-95d2-dc7590a46f8b\") " Oct 30 20:00:20.285941 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:20.285915 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c58d708-c81a-4f9f-95d2-dc7590a46f8b-maas-api-service-ca" pod "5c58d708-c81a-4f9f-95d2-dc7590a46f8b" (UID: "5c58d708-c81a-4f9f-95d2-dc7590a46f8b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:20.288522 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:20.288494 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c58d708-c81a-4f9f-95d2-dc7590a46f8b-kube-api-access-mx7bv" pod "5c58d708-c81a-4f9f-95d2-dc7590a46f8b" (UID: "5c58d708-c81a-4f9f-95d2-dc7590a46f8b"). InnerVolumeSpecName "kube-api-access-mx7bv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:20.386419 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:20.386381 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx7bv\" (UniqueName: \"kubernetes.io/projected/5c58d708-c81a-4f9f-95d2-dc7590a46f8b-kube-api-access-mx7bv\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:20.386419 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:20.386422 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c58d708-c81a-4f9f-95d2-dc7590a46f8b-maas-api-service-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:20.687387 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:20.680590 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d80c7d-29889840-q5c97" Oct 30 20:00:20.687387 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:20.680611 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d80c7d-29889840-q5c97" event={"ID":"875f38d8-03c7-452d-bfd5-e2fc26a738bd","Type":"ContainerDied","Data":"5f2328b0fbc1324c78d649fb7ca4feafe33ea01be4e8ebf35fec31a874604a7e"} Oct 30 20:00:20.687387 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:20.680682 4292 scope.go:122] "RemoveContainer" containerID="1cedffe903b82e20716abe0492a74e893da6b86df3858a670caeb6a64e3e9019" Oct 30 20:00:20.687387 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:20.683091 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889840-dpnvp" event={"ID":"5c58d708-c81a-4f9f-95d2-dc7590a46f8b","Type":"ContainerDied","Data":"00e64b685bd17c6ab70f54134c6622daf0894ae60dbc31af91d6b4fa56540e69"} Oct 30 20:00:20.687387 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:20.683226 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889840-dpnvp" Oct 30 20:00:20.693087 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:20.693056 4292 scope.go:122] "RemoveContainer" containerID="4a74a3b448f16fb70a00d538dc673ba09369742931962ab19d7891976e49cafa" Oct 30 20:00:20.826099 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:20.826073 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889840-6x7xp" Oct 30 20:00:20.891257 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:20.891187 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8c60c3f-f8e7-4980-8823-b573b514da41-kube-api-access-h9xbh\" (UniqueName: \"kubernetes.io/projected/b8c60c3f-f8e7-4980-8823-b573b514da41-kube-api-access-h9xbh\") pod \"b8c60c3f-f8e7-4980-8823-b573b514da41\" (UID: \"b8c60c3f-f8e7-4980-8823-b573b514da41\") " Oct 30 20:00:20.891458 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:20.891392 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8c60c3f-f8e7-4980-8823-b573b514da41-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8c60c3f-f8e7-4980-8823-b573b514da41-maas-api-service-ca\") pod \"b8c60c3f-f8e7-4980-8823-b573b514da41\" (UID: \"b8c60c3f-f8e7-4980-8823-b573b514da41\") " Oct 30 20:00:20.891855 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:20.891777 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8c60c3f-f8e7-4980-8823-b573b514da41-maas-api-service-ca" pod "b8c60c3f-f8e7-4980-8823-b573b514da41" (UID: "b8c60c3f-f8e7-4980-8823-b573b514da41"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:20.893290 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:20.893271 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8c60c3f-f8e7-4980-8823-b573b514da41-kube-api-access-h9xbh" pod "b8c60c3f-f8e7-4980-8823-b573b514da41" (UID: "b8c60c3f-f8e7-4980-8823-b573b514da41"). InnerVolumeSpecName "kube-api-access-h9xbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:20.970798 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:20.970762 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d80c7d-29889840-q5c97"] Oct 30 20:00:20.973862 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:20.973837 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d80c7d-29889840-q5c97"] Oct 30 20:00:20.992806 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:20.992786 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8c60c3f-f8e7-4980-8823-b573b514da41-maas-api-service-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:20.992806 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:20.992807 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9xbh\" (UniqueName: \"kubernetes.io/projected/b8c60c3f-f8e7-4980-8823-b573b514da41-kube-api-access-h9xbh\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:21.086461 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:21.086428 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889840-dpnvp"] Oct 30 20:00:21.088709 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:21.088686 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889840-dpnvp"] Oct 30 20:00:21.692170 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:21.692144 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889840-6x7xp" Oct 30 20:00:21.692671 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:21.692170 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889840-6x7xp" event={"ID":"b8c60c3f-f8e7-4980-8823-b573b514da41","Type":"ContainerDied","Data":"923cd5147318c243f645438670ae6a0124e1db649d9bcab59cce89d3a7640ca9"} Oct 30 20:00:21.692671 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:21.692232 4292 scope.go:122] "RemoveContainer" containerID="964c1b00052909db92392d74432d0f65f24f71d41839185f5eed054b6cb3ac99" Oct 30 20:00:21.860221 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:21.860190 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889840-6x7xp"] Oct 30 20:00:21.864384 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:21.864339 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889840-6x7xp"] Oct 30 20:00:22.602380 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:22.600227 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c58d708-c81a-4f9f-95d2-dc7590a46f8b" path="/var/lib/kubelet/pods/5c58d708-c81a-4f9f-95d2-dc7590a46f8b/volumes" Oct 30 20:00:22.602643 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:22.602468 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="875f38d8-03c7-452d-bfd5-e2fc26a738bd" path="/var/lib/kubelet/pods/875f38d8-03c7-452d-bfd5-e2fc26a738bd/volumes" Oct 30 20:00:22.614369 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:22.603074 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8c60c3f-f8e7-4980-8823-b573b514da41" path="/var/lib/kubelet/pods/b8c60c3f-f8e7-4980-8823-b573b514da41/volumes" Oct 30 20:00:27.706376 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:27.705850 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3f9b9b-66ccc5955-mk88g"] Oct 30 20:00:27.706894 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:27.706547 4292 cpu_manager.go:414] "containerMap: removing container" podUID="0363aa9f-fcfb-4ddc-b950-23441ea7f137" containerName="cleanup" Oct 30 20:00:27.706894 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:27.706569 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0363aa9f-fcfb-4ddc-b950-23441ea7f137" containerName="cleanup" Oct 30 20:00:27.706894 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:27.706590 4292 cpu_manager.go:414] "containerMap: removing container" podUID="b8c60c3f-f8e7-4980-8823-b573b514da41" containerName="cleanup" Oct 30 20:00:27.706894 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:27.706599 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8c60c3f-f8e7-4980-8823-b573b514da41" containerName="cleanup" Oct 30 20:00:27.706894 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:27.706612 4292 cpu_manager.go:414] "containerMap: removing container" podUID="b8c60c3f-f8e7-4980-8823-b573b514da41" containerName="cleanup" Oct 30 20:00:27.706894 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:27.706620 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8c60c3f-f8e7-4980-8823-b573b514da41" containerName="cleanup" Oct 30 20:00:27.706894 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:27.706634 4292 cpu_manager.go:414] "containerMap: removing container" podUID="875f38d8-03c7-452d-bfd5-e2fc26a738bd" containerName="cleanup" Oct 30 20:00:27.706894 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:27.706642 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="875f38d8-03c7-452d-bfd5-e2fc26a738bd" containerName="cleanup" Oct 30 20:00:27.706894 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:27.706650 4292 cpu_manager.go:414] "containerMap: removing container" podUID="875f38d8-03c7-452d-bfd5-e2fc26a738bd" containerName="cleanup" Oct 30 20:00:27.706894 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:27.706658 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="875f38d8-03c7-452d-bfd5-e2fc26a738bd" containerName="cleanup" Oct 30 20:00:27.706894 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:27.706687 4292 cpu_manager.go:414] "containerMap: removing container" podUID="875f38d8-03c7-452d-bfd5-e2fc26a738bd" containerName="cleanup" Oct 30 20:00:27.706894 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:27.706694 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="875f38d8-03c7-452d-bfd5-e2fc26a738bd" containerName="cleanup" Oct 30 20:00:27.706894 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:27.706703 4292 cpu_manager.go:414] "containerMap: removing container" podUID="5c58d708-c81a-4f9f-95d2-dc7590a46f8b" containerName="cleanup" Oct 30 20:00:27.706894 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:27.706710 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c58d708-c81a-4f9f-95d2-dc7590a46f8b" containerName="cleanup" Oct 30 20:00:27.706894 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:27.706723 4292 cpu_manager.go:414] "containerMap: removing container" podUID="5c58d708-c81a-4f9f-95d2-dc7590a46f8b" containerName="cleanup" Oct 30 20:00:27.706894 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:27.706729 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c58d708-c81a-4f9f-95d2-dc7590a46f8b" containerName="cleanup" Oct 30 20:00:27.706894 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:27.706756 4292 cpu_manager.go:414] "containerMap: removing container" podUID="5c58d708-c81a-4f9f-95d2-dc7590a46f8b" containerName="cleanup" Oct 30 20:00:27.706894 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:27.706762 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c58d708-c81a-4f9f-95d2-dc7590a46f8b" containerName="cleanup" Oct 30 20:00:27.706894 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:27.706778 4292 cpu_manager.go:414] "containerMap: removing container" podUID="0363aa9f-fcfb-4ddc-b950-23441ea7f137" containerName="cleanup" Oct 30 20:00:27.706894 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:27.706784 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0363aa9f-fcfb-4ddc-b950-23441ea7f137" containerName="cleanup" Oct 30 20:00:27.706894 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:27.706894 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8c60c3f-f8e7-4980-8823-b573b514da41" containerName="cleanup" Oct 30 20:00:27.707985 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:27.706909 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c58d708-c81a-4f9f-95d2-dc7590a46f8b" containerName="cleanup" Oct 30 20:00:27.707985 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:27.706920 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="0363aa9f-fcfb-4ddc-b950-23441ea7f137" containerName="cleanup" Oct 30 20:00:27.707985 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:27.706929 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="0363aa9f-fcfb-4ddc-b950-23441ea7f137" containerName="cleanup" Oct 30 20:00:27.707985 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:27.706937 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="0363aa9f-fcfb-4ddc-b950-23441ea7f137" containerName="cleanup" Oct 30 20:00:27.707985 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:27.706948 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8c60c3f-f8e7-4980-8823-b573b514da41" containerName="cleanup" Oct 30 20:00:27.707985 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:27.706956 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8c60c3f-f8e7-4980-8823-b573b514da41" containerName="cleanup" Oct 30 20:00:27.707985 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:27.706967 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="875f38d8-03c7-452d-bfd5-e2fc26a738bd" containerName="cleanup" Oct 30 20:00:27.707985 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:27.706975 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="875f38d8-03c7-452d-bfd5-e2fc26a738bd" containerName="cleanup" Oct 30 20:00:27.707985 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:27.706986 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="875f38d8-03c7-452d-bfd5-e2fc26a738bd" containerName="cleanup" Oct 30 20:00:27.707985 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:27.706995 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c58d708-c81a-4f9f-95d2-dc7590a46f8b" containerName="cleanup" Oct 30 20:00:27.707985 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:27.707002 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c58d708-c81a-4f9f-95d2-dc7590a46f8b" containerName="cleanup" Oct 30 20:00:27.718661 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:27.717568 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f9b9b-66ccc5955-mk88g" Oct 30 20:00:27.776374 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:27.773693 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-3f9b9b\"" Oct 30 20:00:27.776374 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:27.774435 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-3f9b9b-dockercfg-q49pj\"" Oct 30 20:00:27.776374 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:27.775152 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3f9b9b-66ccc5955-mk88g"] Oct 30 20:00:27.831278 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:27.822002 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fabac46-c08c-45bf-9cfe-b9b139aa4ebc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3f9b9b-66ccc5955-mk88g\" (UID: \"8fabac46-c08c-45bf-9cfe-b9b139aa4ebc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f9b9b-66ccc5955-mk88g" Oct 30 20:00:27.831278 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:27.822116 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnkhm\" (UniqueName: \"kubernetes.io/projected/8fabac46-c08c-45bf-9cfe-b9b139aa4ebc-kube-api-access-hnkhm\") pod \"payload-pre-processing-e2e-ipp-b-3f9b9b-66ccc5955-mk88g\" (UID: \"8fabac46-c08c-45bf-9cfe-b9b139aa4ebc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f9b9b-66ccc5955-mk88g" Oct 30 20:00:27.923099 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:27.923050 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fabac46-c08c-45bf-9cfe-b9b139aa4ebc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3f9b9b-66ccc5955-mk88g\" (UID: \"8fabac46-c08c-45bf-9cfe-b9b139aa4ebc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f9b9b-66ccc5955-mk88g" Oct 30 20:00:27.923473 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:27.923451 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnkhm\" (UniqueName: \"kubernetes.io/projected/8fabac46-c08c-45bf-9cfe-b9b139aa4ebc-kube-api-access-hnkhm\") pod \"payload-pre-processing-e2e-ipp-b-3f9b9b-66ccc5955-mk88g\" (UID: \"8fabac46-c08c-45bf-9cfe-b9b139aa4ebc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f9b9b-66ccc5955-mk88g" Oct 30 20:00:27.928407 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:27.925273 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fabac46-c08c-45bf-9cfe-b9b139aa4ebc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3f9b9b-66ccc5955-mk88g\" (UID: \"8fabac46-c08c-45bf-9cfe-b9b139aa4ebc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f9b9b-66ccc5955-mk88g" Oct 30 20:00:27.992086 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:27.992000 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnkhm\" (UniqueName: \"kubernetes.io/projected/8fabac46-c08c-45bf-9cfe-b9b139aa4ebc-kube-api-access-hnkhm\") pod \"payload-pre-processing-e2e-ipp-b-3f9b9b-66ccc5955-mk88g\" (UID: \"8fabac46-c08c-45bf-9cfe-b9b139aa4ebc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f9b9b-66ccc5955-mk88g" Oct 30 20:00:28.037130 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:28.037102 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f9b9b-66ccc5955-mk88g" Oct 30 20:00:28.440106 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:28.440074 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3f9b9b-66ccc5955-mk88g"] Oct 30 20:00:28.440411 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:00:28.440385 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fabac46_c08c_45bf_9cfe_b9b139aa4ebc.slice/crio-a37eef9c655f70bd77511ce8af559d93af2434524ac7c9e2cadf6874899f049f WatchSource:0}: Error finding container a37eef9c655f70bd77511ce8af559d93af2434524ac7c9e2cadf6874899f049f: Status 404 returned error can't find the container with id a37eef9c655f70bd77511ce8af559d93af2434524ac7c9e2cadf6874899f049f Oct 30 20:00:28.744146 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:28.744056 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f9b9b-66ccc5955-mk88g" event={"ID":"8fabac46-c08c-45bf-9cfe-b9b139aa4ebc","Type":"ContainerStarted","Data":"9fd95aa15059c17d7ef251dd5aacdda4dcee22c5a5822c86a7941bfc828f2df5"} Oct 30 20:00:28.744146 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:28.744141 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f9b9b-66ccc5955-mk88g" event={"ID":"8fabac46-c08c-45bf-9cfe-b9b139aa4ebc","Type":"ContainerStarted","Data":"a37eef9c655f70bd77511ce8af559d93af2434524ac7c9e2cadf6874899f049f"} Oct 30 20:00:28.744604 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:28.744238 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f9b9b-66ccc5955-mk88g" Oct 30 20:00:28.876419 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:28.876320 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f9b9b-66ccc5955-mk88g" podStartSLOduration=1.876301674 podStartE2EDuration="1.876301674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:28.870849661 +0000 UTC m=+2132.389165120" watchObservedRunningTime="2026-10-30 20:00:28.876301674 +0000 UTC m=+2132.394617133" Oct 30 20:00:33.895234 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:33.895186 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9"] Oct 30 20:00:33.895732 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:33.895716 4292 cpu_manager.go:414] "containerMap: removing container" podUID="b8c60c3f-f8e7-4980-8823-b573b514da41" containerName="cleanup" Oct 30 20:00:33.895780 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:33.895733 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8c60c3f-f8e7-4980-8823-b573b514da41" containerName="cleanup" Oct 30 20:00:33.895780 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:33.895770 4292 cpu_manager.go:414] "containerMap: removing container" podUID="0363aa9f-fcfb-4ddc-b950-23441ea7f137" containerName="cleanup" Oct 30 20:00:33.895780 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:33.895776 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0363aa9f-fcfb-4ddc-b950-23441ea7f137" containerName="cleanup" Oct 30 20:00:33.900197 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:33.900179 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9" Oct 30 20:00:33.937290 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:33.936202 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61b24a37-2664-40ff-a96e-58f2d44e1045-istio-podinfo\") pod \"e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9\" (UID: \"61b24a37-2664-40ff-a96e-58f2d44e1045\") " pod="openshift-ingress/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9" Oct 30 20:00:33.937290 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:33.936282 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61b24a37-2664-40ff-a96e-58f2d44e1045-istio-data\") pod \"e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9\" (UID: \"61b24a37-2664-40ff-a96e-58f2d44e1045\") " pod="openshift-ingress/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9" Oct 30 20:00:33.937290 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:33.936320 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61b24a37-2664-40ff-a96e-58f2d44e1045-credential-socket\") pod \"e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9\" (UID: \"61b24a37-2664-40ff-a96e-58f2d44e1045\") " pod="openshift-ingress/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9" Oct 30 20:00:33.937290 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:33.936435 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61b24a37-2664-40ff-a96e-58f2d44e1045-istio-token\") pod \"e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9\" (UID: \"61b24a37-2664-40ff-a96e-58f2d44e1045\") " pod="openshift-ingress/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9" Oct 30 20:00:33.937290 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:33.936470 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61b24a37-2664-40ff-a96e-58f2d44e1045-istio-envoy\") pod \"e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9\" (UID: \"61b24a37-2664-40ff-a96e-58f2d44e1045\") " pod="openshift-ingress/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9" Oct 30 20:00:33.937290 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:33.936523 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk5p9\" (UniqueName: \"kubernetes.io/projected/61b24a37-2664-40ff-a96e-58f2d44e1045-kube-api-access-tk5p9\") pod \"e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9\" (UID: \"61b24a37-2664-40ff-a96e-58f2d44e1045\") " pod="openshift-ingress/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9" Oct 30 20:00:33.937290 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:33.936569 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61b24a37-2664-40ff-a96e-58f2d44e1045-workload-certs\") pod \"e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9\" (UID: \"61b24a37-2664-40ff-a96e-58f2d44e1045\") " pod="openshift-ingress/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9" Oct 30 20:00:33.937290 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:33.936606 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61b24a37-2664-40ff-a96e-58f2d44e1045-workload-socket\") pod \"e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9\" (UID: \"61b24a37-2664-40ff-a96e-58f2d44e1045\") " pod="openshift-ingress/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9" Oct 30 20:00:33.937290 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:33.936696 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61b24a37-2664-40ff-a96e-58f2d44e1045-istiod-ca-cert\") pod \"e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9\" (UID: \"61b24a37-2664-40ff-a96e-58f2d44e1045\") " pod="openshift-ingress/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9" Oct 30 20:00:33.963458 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:33.958540 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-2d2ca9-openshift-default-dockercfg-h6thq\"" Oct 30 20:00:34.138193 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:34.123063 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61b24a37-2664-40ff-a96e-58f2d44e1045-istio-token\") pod \"e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9\" (UID: \"61b24a37-2664-40ff-a96e-58f2d44e1045\") " pod="openshift-ingress/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9" Oct 30 20:00:34.138193 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:34.123153 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61b24a37-2664-40ff-a96e-58f2d44e1045-istio-envoy\") pod \"e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9\" (UID: \"61b24a37-2664-40ff-a96e-58f2d44e1045\") " pod="openshift-ingress/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9" Oct 30 20:00:34.138193 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:34.123204 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk5p9\" (UniqueName: \"kubernetes.io/projected/61b24a37-2664-40ff-a96e-58f2d44e1045-kube-api-access-tk5p9\") pod \"e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9\" (UID: \"61b24a37-2664-40ff-a96e-58f2d44e1045\") " pod="openshift-ingress/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9" Oct 30 20:00:34.138193 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:34.123261 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61b24a37-2664-40ff-a96e-58f2d44e1045-workload-certs\") pod \"e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9\" (UID: \"61b24a37-2664-40ff-a96e-58f2d44e1045\") " pod="openshift-ingress/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9" Oct 30 20:00:34.138193 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:34.123305 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61b24a37-2664-40ff-a96e-58f2d44e1045-workload-socket\") pod \"e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9\" (UID: \"61b24a37-2664-40ff-a96e-58f2d44e1045\") " pod="openshift-ingress/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9" Oct 30 20:00:34.138193 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:34.123440 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61b24a37-2664-40ff-a96e-58f2d44e1045-istiod-ca-cert\") pod \"e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9\" (UID: \"61b24a37-2664-40ff-a96e-58f2d44e1045\") " pod="openshift-ingress/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9" Oct 30 20:00:34.138193 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:34.123535 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61b24a37-2664-40ff-a96e-58f2d44e1045-istio-podinfo\") pod \"e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9\" (UID: \"61b24a37-2664-40ff-a96e-58f2d44e1045\") " pod="openshift-ingress/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9" Oct 30 20:00:34.147437 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:34.140019 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61b24a37-2664-40ff-a96e-58f2d44e1045-istio-envoy\") pod \"e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9\" (UID: \"61b24a37-2664-40ff-a96e-58f2d44e1045\") " pod="openshift-ingress/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9" Oct 30 20:00:34.147437 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:34.142603 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61b24a37-2664-40ff-a96e-58f2d44e1045-istio-data\") pod \"e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9\" (UID: \"61b24a37-2664-40ff-a96e-58f2d44e1045\") " pod="openshift-ingress/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9" Oct 30 20:00:34.147437 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:34.142665 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61b24a37-2664-40ff-a96e-58f2d44e1045-credential-socket\") pod \"e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9\" (UID: \"61b24a37-2664-40ff-a96e-58f2d44e1045\") " pod="openshift-ingress/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9" Oct 30 20:00:34.147437 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:34.143181 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61b24a37-2664-40ff-a96e-58f2d44e1045-credential-socket\") pod \"e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9\" (UID: \"61b24a37-2664-40ff-a96e-58f2d44e1045\") " pod="openshift-ingress/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9" Oct 30 20:00:34.147437 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:34.143436 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61b24a37-2664-40ff-a96e-58f2d44e1045-workload-socket\") pod \"e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9\" (UID: \"61b24a37-2664-40ff-a96e-58f2d44e1045\") " pod="openshift-ingress/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9" Oct 30 20:00:34.147437 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:34.143650 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61b24a37-2664-40ff-a96e-58f2d44e1045-workload-certs\") pod \"e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9\" (UID: \"61b24a37-2664-40ff-a96e-58f2d44e1045\") " pod="openshift-ingress/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9" Oct 30 20:00:34.147437 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:34.144395 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61b24a37-2664-40ff-a96e-58f2d44e1045-istiod-ca-cert\") pod \"e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9\" (UID: \"61b24a37-2664-40ff-a96e-58f2d44e1045\") " pod="openshift-ingress/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9" Oct 30 20:00:34.147437 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:34.144657 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61b24a37-2664-40ff-a96e-58f2d44e1045-istio-data\") pod \"e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9\" (UID: \"61b24a37-2664-40ff-a96e-58f2d44e1045\") " pod="openshift-ingress/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9" Oct 30 20:00:34.158895 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:34.158680 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61b24a37-2664-40ff-a96e-58f2d44e1045-istio-podinfo\") pod \"e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9\" (UID: \"61b24a37-2664-40ff-a96e-58f2d44e1045\") " pod="openshift-ingress/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9" Oct 30 20:00:34.158895 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:34.158753 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9"] Oct 30 20:00:34.204155 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:34.204115 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk5p9\" (UniqueName: \"kubernetes.io/projected/61b24a37-2664-40ff-a96e-58f2d44e1045-kube-api-access-tk5p9\") pod \"e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9\" (UID: \"61b24a37-2664-40ff-a96e-58f2d44e1045\") " pod="openshift-ingress/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9" Oct 30 20:00:34.204283 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:34.204194 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61b24a37-2664-40ff-a96e-58f2d44e1045-istio-token\") pod \"e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9\" (UID: \"61b24a37-2664-40ff-a96e-58f2d44e1045\") " pod="openshift-ingress/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9" Oct 30 20:00:34.259222 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:34.259187 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9" Oct 30 20:00:34.774342 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:34.774313 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9"] Oct 30 20:00:34.776048 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:00:34.776024 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61b24a37_2664_40ff_a96e_58f2d44e1045.slice/crio-f862dc2f7a363d68faa0fb1c599948de14b159ac883ba5fca71f1e114505eb11 WatchSource:0}: Error finding container f862dc2f7a363d68faa0fb1c599948de14b159ac883ba5fca71f1e114505eb11: Status 404 returned error can't find the container with id f862dc2f7a363d68faa0fb1c599948de14b159ac883ba5fca71f1e114505eb11 Oct 30 20:00:34.779058 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:34.779017 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:00:34.779182 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:34.779140 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:00:34.779256 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:34.779189 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:00:34.786835 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:34.786812 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9" event={"ID":"61b24a37-2664-40ff-a96e-58f2d44e1045","Type":"ContainerStarted","Data":"f862dc2f7a363d68faa0fb1c599948de14b159ac883ba5fca71f1e114505eb11"} Oct 30 20:00:34.788545 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:34.788524 4292 generic.go:356] "Generic (PLEG): container finished" podID="554fd8be-cebc-4035-840e-e377169383e5" containerID="4d37c3a61cf245489e3daf66c689410e3d7155c5ed1263cb202415cdd7d6d768" exitCode=0 Oct 30 20:00:34.788668 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:34.788604 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6" event={"ID":"554fd8be-cebc-4035-840e-e377169383e5","Type":"ContainerDied","Data":"4d37c3a61cf245489e3daf66c689410e3d7155c5ed1263cb202415cdd7d6d768"} Oct 30 20:00:34.789090 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:34.789073 4292 scope.go:122] "RemoveContainer" containerID="4d37c3a61cf245489e3daf66c689410e3d7155c5ed1263cb202415cdd7d6d768" Oct 30 20:00:34.790617 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:34.790587 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:00:34.790713 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:34.790671 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:00:34.790771 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:34.790715 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:00:35.800639 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:35.800601 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6" event={"ID":"554fd8be-cebc-4035-840e-e377169383e5","Type":"ContainerStarted","Data":"05237ba0c0e7a3109e8cb1ccde90bfc76424465f898d28d34e12224021a3f57b"} Oct 30 20:00:35.804196 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:35.804163 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9" event={"ID":"61b24a37-2664-40ff-a96e-58f2d44e1045","Type":"ContainerStarted","Data":"de7e331af7e302ee6ae20cd723fd13e3b790e95de0a254d4bba55c5c01a7b89b"} Oct 30 20:00:36.263432 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:36.260731 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9" Oct 30 20:00:36.273688 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:36.273657 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9" Oct 30 20:00:36.405664 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:36.405613 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9" podStartSLOduration=3.405598482 podStartE2EDuration="3.405598482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:36.110464589 +0000 UTC m=+2139.628780070" watchObservedRunningTime="2026-10-30 20:00:36.405598482 +0000 UTC m=+2139.923913939" Oct 30 20:00:36.728775 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:36.728740 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6" Oct 30 20:00:36.728983 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:36.728906 4292 patch_prober.go:29] interesting pod/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 20:00:36.728983 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:36.728959 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6" podUID="554fd8be-cebc-4035-840e-e377169383e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 20:00:36.809512 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:36.809481 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9" Oct 30 20:00:36.810475 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:36.810454 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9" Oct 30 20:00:37.729224 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:37.729190 4292 patch_prober.go:29] interesting pod/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 20:00:37.729439 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:37.729245 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6" podUID="554fd8be-cebc-4035-840e-e377169383e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 20:00:38.729064 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:38.729033 4292 patch_prober.go:29] interesting pod/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 20:00:38.729519 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:38.729084 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6" podUID="554fd8be-cebc-4035-840e-e377169383e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 20:00:39.729257 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:39.729221 4292 patch_prober.go:29] interesting pod/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 20:00:39.729701 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:39.729285 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6" podUID="554fd8be-cebc-4035-840e-e377169383e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 20:00:39.769584 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:39.769556 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f9b9b-66ccc5955-mk88g" Oct 30 20:00:40.729675 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:40.729645 4292 patch_prober.go:29] interesting pod/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 20:00:40.730133 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:40.729699 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6" podUID="554fd8be-cebc-4035-840e-e377169383e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 20:00:41.733954 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:41.733922 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6" Oct 30 20:00:41.734436 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:41.734341 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6" Oct 30 20:00:41.735003 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:41.734985 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6" Oct 30 20:00:43.228477 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:00:43.228443 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:00:57.015443 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:00:57.015412 4292 scope.go:122] "RemoveContainer" containerID="0c45484666f573a2a3ef6aac32d0a55a797e6b40a1ebc4eb22af939746ab332b" Oct 30 20:01:06.890106 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:06.890073 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ffe14b-76d78847cb-gq7tv"] Oct 30 20:01:11.890684 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:11.890641 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-ffe14b-76d78847cb-gq7tv" podUID="c3759813-8a90-4e72-b94d-86634ee99202" containerName="payload-processing" containerID="cri-o://35052d8644c74695b12a245878c333e6a4646c02e8fa90c160fc93ca60b052b9" gracePeriod=25 Oct 30 20:01:12.137779 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:12.137749 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ffe14b-76d78847cb-gq7tv" Oct 30 20:01:12.265063 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:12.264989 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3759813-8a90-4e72-b94d-86634ee99202-kube-api-access-vhplf\" (UniqueName: \"kubernetes.io/projected/c3759813-8a90-4e72-b94d-86634ee99202-kube-api-access-vhplf\") pod \"c3759813-8a90-4e72-b94d-86634ee99202\" (UID: \"c3759813-8a90-4e72-b94d-86634ee99202\") " Oct 30 20:01:12.265206 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:12.265083 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3759813-8a90-4e72-b94d-86634ee99202-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3759813-8a90-4e72-b94d-86634ee99202-plugins-config-volume\") pod \"c3759813-8a90-4e72-b94d-86634ee99202\" (UID: \"c3759813-8a90-4e72-b94d-86634ee99202\") " Oct 30 20:01:12.265498 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:12.265472 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3759813-8a90-4e72-b94d-86634ee99202-plugins-config-volume" pod "c3759813-8a90-4e72-b94d-86634ee99202" (UID: "c3759813-8a90-4e72-b94d-86634ee99202"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:01:12.267248 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:12.267223 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3759813-8a90-4e72-b94d-86634ee99202-kube-api-access-vhplf" pod "c3759813-8a90-4e72-b94d-86634ee99202" (UID: "c3759813-8a90-4e72-b94d-86634ee99202"). InnerVolumeSpecName "kube-api-access-vhplf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:12.366574 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:12.366541 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3759813-8a90-4e72-b94d-86634ee99202-plugins-config-volume\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:12.366574 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:12.366568 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhplf\" (UniqueName: \"kubernetes.io/projected/c3759813-8a90-4e72-b94d-86634ee99202-kube-api-access-vhplf\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:13.010542 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:13.010450 4292 generic.go:356] "Generic (PLEG): container finished" podID="c3759813-8a90-4e72-b94d-86634ee99202" containerID="35052d8644c74695b12a245878c333e6a4646c02e8fa90c160fc93ca60b052b9" exitCode=0 Oct 30 20:01:13.010542 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:13.010508 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ffe14b-76d78847cb-gq7tv" Oct 30 20:01:13.010542 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:13.010525 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ffe14b-76d78847cb-gq7tv" event={"ID":"c3759813-8a90-4e72-b94d-86634ee99202","Type":"ContainerDied","Data":"35052d8644c74695b12a245878c333e6a4646c02e8fa90c160fc93ca60b052b9"} Oct 30 20:01:13.011064 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:13.010580 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ffe14b-76d78847cb-gq7tv" event={"ID":"c3759813-8a90-4e72-b94d-86634ee99202","Type":"ContainerDied","Data":"cde5076a0bc630def4d0bc4e3bf258eb940c74c5b70a8018126db7bc850b8ce8"} Oct 30 20:01:13.011064 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:13.010602 4292 scope.go:122] "RemoveContainer" containerID="35052d8644c74695b12a245878c333e6a4646c02e8fa90c160fc93ca60b052b9" Oct 30 20:01:13.019492 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:13.019458 4292 scope.go:122] "RemoveContainer" containerID="35052d8644c74695b12a245878c333e6a4646c02e8fa90c160fc93ca60b052b9" Oct 30 20:01:13.019754 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:01:13.019733 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35052d8644c74695b12a245878c333e6a4646c02e8fa90c160fc93ca60b052b9\": container with ID starting with 35052d8644c74695b12a245878c333e6a4646c02e8fa90c160fc93ca60b052b9 not found: ID does not exist" containerID="35052d8644c74695b12a245878c333e6a4646c02e8fa90c160fc93ca60b052b9" Oct 30 20:01:13.019828 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:13.019761 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35052d8644c74695b12a245878c333e6a4646c02e8fa90c160fc93ca60b052b9"} err="failed to get container status \"35052d8644c74695b12a245878c333e6a4646c02e8fa90c160fc93ca60b052b9\": rpc error: code = NotFound desc = could not find container \"35052d8644c74695b12a245878c333e6a4646c02e8fa90c160fc93ca60b052b9\": container with ID starting with 35052d8644c74695b12a245878c333e6a4646c02e8fa90c160fc93ca60b052b9 not found: ID does not exist" Oct 30 20:01:13.182944 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:13.182906 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ffe14b-76d78847cb-gq7tv"] Oct 30 20:01:13.192766 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:13.192741 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ffe14b-76d78847cb-gq7tv"] Oct 30 20:01:13.223211 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:01:13.223174 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:01:14.582215 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:14.582183 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3759813-8a90-4e72-b94d-86634ee99202" path="/var/lib/kubelet/pods/c3759813-8a90-4e72-b94d-86634ee99202/volumes" Oct 30 20:01:17.521484 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:17.521428 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-77a92c-65f6f789f-rxgqj"] Oct 30 20:01:17.522863 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:17.522833 4292 cpu_manager.go:414] "containerMap: removing container" podUID="c3759813-8a90-4e72-b94d-86634ee99202" containerName="payload-processing" Oct 30 20:01:17.523028 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:17.523012 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3759813-8a90-4e72-b94d-86634ee99202" containerName="payload-processing" Oct 30 20:01:17.523284 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:17.523266 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3759813-8a90-4e72-b94d-86634ee99202" containerName="payload-processing" Oct 30 20:01:17.527501 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:17.527479 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77a92c-65f6f789f-rxgqj" Oct 30 20:01:17.584905 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:17.584865 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-77a92c-65f6f789f-rxgqj"] Oct 30 20:01:17.626679 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:17.626648 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-77a92c\"" Oct 30 20:01:17.629499 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:17.629470 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfv8d\" (UniqueName: \"kubernetes.io/projected/5f6986e0-8689-412d-bf77-450b85b0254d-kube-api-access-zfv8d\") pod \"payload-pre-processing-e2e-ipp-b-77a92c-65f6f789f-rxgqj\" (UID: \"5f6986e0-8689-412d-bf77-450b85b0254d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77a92c-65f6f789f-rxgqj" Oct 30 20:01:17.629735 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:17.629715 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f6986e0-8689-412d-bf77-450b85b0254d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-77a92c-65f6f789f-rxgqj\" (UID: \"5f6986e0-8689-412d-bf77-450b85b0254d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77a92c-65f6f789f-rxgqj" Oct 30 20:01:17.642013 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:17.633275 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-77a92c-dockercfg-8jm4f\"" Oct 30 20:01:17.645849 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:17.645822 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-77a92c-64d9fc577b-pbdb5"] Oct 30 20:01:17.649454 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:17.649428 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-77a92c-64d9fc577b-pbdb5" Oct 30 20:01:17.706780 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:17.706747 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-77a92c-64d9fc577b-pbdb5"] Oct 30 20:01:17.731050 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:17.731016 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f6986e0-8689-412d-bf77-450b85b0254d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-77a92c-65f6f789f-rxgqj\" (UID: \"5f6986e0-8689-412d-bf77-450b85b0254d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77a92c-65f6f789f-rxgqj" Oct 30 20:01:17.731467 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:17.731436 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9211eb29-a409-473e-b654-09612e213dc7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-77a92c-64d9fc577b-pbdb5\" (UID: \"9211eb29-a409-473e-b654-09612e213dc7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-77a92c-64d9fc577b-pbdb5" Oct 30 20:01:17.731586 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:17.731528 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nh2q\" (UniqueName: \"kubernetes.io/projected/9211eb29-a409-473e-b654-09612e213dc7-kube-api-access-8nh2q\") pod \"payload-processing-e2e-ipp-b-77a92c-64d9fc577b-pbdb5\" (UID: \"9211eb29-a409-473e-b654-09612e213dc7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-77a92c-64d9fc577b-pbdb5" Oct 30 20:01:17.731650 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:17.731581 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfv8d\" (UniqueName: \"kubernetes.io/projected/5f6986e0-8689-412d-bf77-450b85b0254d-kube-api-access-zfv8d\") pod \"payload-pre-processing-e2e-ipp-b-77a92c-65f6f789f-rxgqj\" (UID: \"5f6986e0-8689-412d-bf77-450b85b0254d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77a92c-65f6f789f-rxgqj" Oct 30 20:01:17.732108 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:17.732080 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f6986e0-8689-412d-bf77-450b85b0254d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-77a92c-65f6f789f-rxgqj\" (UID: \"5f6986e0-8689-412d-bf77-450b85b0254d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77a92c-65f6f789f-rxgqj" Oct 30 20:01:17.794475 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:17.794415 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfv8d\" (UniqueName: \"kubernetes.io/projected/5f6986e0-8689-412d-bf77-450b85b0254d-kube-api-access-zfv8d\") pod \"payload-pre-processing-e2e-ipp-b-77a92c-65f6f789f-rxgqj\" (UID: \"5f6986e0-8689-412d-bf77-450b85b0254d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77a92c-65f6f789f-rxgqj" Oct 30 20:01:17.832314 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:17.832281 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nh2q\" (UniqueName: \"kubernetes.io/projected/9211eb29-a409-473e-b654-09612e213dc7-kube-api-access-8nh2q\") pod \"payload-processing-e2e-ipp-b-77a92c-64d9fc577b-pbdb5\" (UID: \"9211eb29-a409-473e-b654-09612e213dc7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-77a92c-64d9fc577b-pbdb5" Oct 30 20:01:17.832498 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:17.832480 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9211eb29-a409-473e-b654-09612e213dc7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-77a92c-64d9fc577b-pbdb5\" (UID: \"9211eb29-a409-473e-b654-09612e213dc7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-77a92c-64d9fc577b-pbdb5" Oct 30 20:01:17.833128 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:17.833110 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9211eb29-a409-473e-b654-09612e213dc7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-77a92c-64d9fc577b-pbdb5\" (UID: \"9211eb29-a409-473e-b654-09612e213dc7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-77a92c-64d9fc577b-pbdb5" Oct 30 20:01:17.895051 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:17.895019 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nh2q\" (UniqueName: \"kubernetes.io/projected/9211eb29-a409-473e-b654-09612e213dc7-kube-api-access-8nh2q\") pod \"payload-processing-e2e-ipp-b-77a92c-64d9fc577b-pbdb5\" (UID: \"9211eb29-a409-473e-b654-09612e213dc7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-77a92c-64d9fc577b-pbdb5" Oct 30 20:01:17.921872 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:17.921838 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77a92c-65f6f789f-rxgqj" Oct 30 20:01:17.964620 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:17.964583 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-77a92c-64d9fc577b-pbdb5" Oct 30 20:01:18.297538 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:18.297509 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-77a92c-65f6f789f-rxgqj"] Oct 30 20:01:18.297931 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:01:18.297907 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f6986e0_8689_412d_bf77_450b85b0254d.slice/crio-fd3e6811b36c3d037cc5636dbd0223f5a0972cca95e7f1f2785da0007b2e1371 WatchSource:0}: Error finding container fd3e6811b36c3d037cc5636dbd0223f5a0972cca95e7f1f2785da0007b2e1371: Status 404 returned error can't find the container with id fd3e6811b36c3d037cc5636dbd0223f5a0972cca95e7f1f2785da0007b2e1371 Oct 30 20:01:18.363380 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:18.363329 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-77a92c-64d9fc577b-pbdb5"] Oct 30 20:01:18.366297 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:01:18.366271 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9211eb29_a409_473e_b654_09612e213dc7.slice/crio-ee5241a4b1e0503e2962f74db311d7d141cb7f84bdc35df28c6aa234a7b65c89 WatchSource:0}: Error finding container ee5241a4b1e0503e2962f74db311d7d141cb7f84bdc35df28c6aa234a7b65c89: Status 404 returned error can't find the container with id ee5241a4b1e0503e2962f74db311d7d141cb7f84bdc35df28c6aa234a7b65c89 Oct 30 20:01:19.075926 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:19.075888 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77a92c-65f6f789f-rxgqj" event={"ID":"5f6986e0-8689-412d-bf77-450b85b0254d","Type":"ContainerStarted","Data":"4c155d2a1b742e13cc2ad4e79fb5795ad6f054fc08a6ea7000ce86bb4277f995"} Oct 30 20:01:19.076428 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:19.075955 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77a92c-65f6f789f-rxgqj" Oct 30 20:01:19.076428 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:19.075982 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77a92c-65f6f789f-rxgqj" event={"ID":"5f6986e0-8689-412d-bf77-450b85b0254d","Type":"ContainerStarted","Data":"fd3e6811b36c3d037cc5636dbd0223f5a0972cca95e7f1f2785da0007b2e1371"} Oct 30 20:01:19.077612 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:19.077582 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-77a92c-64d9fc577b-pbdb5" event={"ID":"9211eb29-a409-473e-b654-09612e213dc7","Type":"ContainerStarted","Data":"fa5ece7c50fec54ea2d2673bed0fc109a9372d9a38ca83c11fcede402811dde1"} Oct 30 20:01:19.077735 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:19.077629 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-77a92c-64d9fc577b-pbdb5" event={"ID":"9211eb29-a409-473e-b654-09612e213dc7","Type":"ContainerStarted","Data":"ee5241a4b1e0503e2962f74db311d7d141cb7f84bdc35df28c6aa234a7b65c89"} Oct 30 20:01:19.077735 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:19.077713 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-77a92c-64d9fc577b-pbdb5" Oct 30 20:01:19.202982 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:19.202935 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77a92c-65f6f789f-rxgqj" podStartSLOduration=2.202920246 podStartE2EDuration="2.202920246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:19.200492241 +0000 UTC m=+2182.718807701" watchObservedRunningTime="2026-10-30 20:01:19.202920246 +0000 UTC m=+2182.721235702" Oct 30 20:01:19.343006 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:19.342869 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-77a92c-64d9fc577b-pbdb5" podStartSLOduration=2.342833903 podStartE2EDuration="2.342833903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:19.335497406 +0000 UTC m=+2182.853812865" watchObservedRunningTime="2026-10-30 20:01:19.342833903 +0000 UTC m=+2182.861149362" Oct 30 20:01:30.088899 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:30.088868 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-77a92c-64d9fc577b-pbdb5" Oct 30 20:01:30.089471 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:30.088940 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77a92c-65f6f789f-rxgqj" Oct 30 20:01:33.159454 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:33.159413 4292 generic.go:356] "Generic (PLEG): container finished" podID="cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe" containerID="6b0c868cf678f8da347762ae591ea92b7ead5baa887c9be3d21cb5dc98c83c61" exitCode=0 Oct 30 20:01:33.159894 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:33.159485 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr" event={"ID":"cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe","Type":"ContainerDied","Data":"6b0c868cf678f8da347762ae591ea92b7ead5baa887c9be3d21cb5dc98c83c61"} Oct 30 20:01:33.159894 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:33.159887 4292 scope.go:122] "RemoveContainer" containerID="6b0c868cf678f8da347762ae591ea92b7ead5baa887c9be3d21cb5dc98c83c61" Oct 30 20:01:33.161295 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:33.161269 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:01:33.161381 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:33.161346 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:01:33.161423 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:33.161395 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:01:34.179400 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:34.179324 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr" event={"ID":"cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe","Type":"ContainerStarted","Data":"9f19d7f1745cd961b77bab3ee2f4be2bd854c951b726fdc08ff73058a6c196fd"} Oct 30 20:01:34.374990 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:34.374925 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaa6d2-74d8499dd4-wfpmb"] Oct 30 20:01:34.378153 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:34.378129 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaa6d2-74d8499dd4-wfpmb" Oct 30 20:01:34.436215 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:34.436142 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaa6d2-74d8499dd4-wfpmb"] Oct 30 20:01:34.436921 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:34.436886 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-eaa6d2-serving-cert\"" Oct 30 20:01:34.480194 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:34.480145 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/510eee0c-6b42-4e13-83c3-5d91a98eb7aa-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-eaa6d2-74d8499dd4-wfpmb\" (UID: \"510eee0c-6b42-4e13-83c3-5d91a98eb7aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaa6d2-74d8499dd4-wfpmb" Oct 30 20:01:34.480414 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:34.480199 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/510eee0c-6b42-4e13-83c3-5d91a98eb7aa-maas-api-tls\") pod \"maas-api-e2e-ipp-a-eaa6d2-74d8499dd4-wfpmb\" (UID: \"510eee0c-6b42-4e13-83c3-5d91a98eb7aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaa6d2-74d8499dd4-wfpmb" Oct 30 20:01:34.480471 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:34.480417 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/510eee0c-6b42-4e13-83c3-5d91a98eb7aa-metrics-certs\") pod \"maas-api-e2e-ipp-a-eaa6d2-74d8499dd4-wfpmb\" (UID: \"510eee0c-6b42-4e13-83c3-5d91a98eb7aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaa6d2-74d8499dd4-wfpmb" Oct 30 20:01:34.480471 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:34.480455 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvfq6\" (UniqueName: \"kubernetes.io/projected/510eee0c-6b42-4e13-83c3-5d91a98eb7aa-kube-api-access-nvfq6\") pod \"maas-api-e2e-ipp-a-eaa6d2-74d8499dd4-wfpmb\" (UID: \"510eee0c-6b42-4e13-83c3-5d91a98eb7aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaa6d2-74d8499dd4-wfpmb" Oct 30 20:01:34.581948 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:34.581914 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/510eee0c-6b42-4e13-83c3-5d91a98eb7aa-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-eaa6d2-74d8499dd4-wfpmb\" (UID: \"510eee0c-6b42-4e13-83c3-5d91a98eb7aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaa6d2-74d8499dd4-wfpmb" Oct 30 20:01:34.582129 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:34.581959 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/510eee0c-6b42-4e13-83c3-5d91a98eb7aa-maas-api-tls\") pod \"maas-api-e2e-ipp-a-eaa6d2-74d8499dd4-wfpmb\" (UID: \"510eee0c-6b42-4e13-83c3-5d91a98eb7aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaa6d2-74d8499dd4-wfpmb" Oct 30 20:01:34.582129 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:34.582060 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/510eee0c-6b42-4e13-83c3-5d91a98eb7aa-metrics-certs\") pod \"maas-api-e2e-ipp-a-eaa6d2-74d8499dd4-wfpmb\" (UID: \"510eee0c-6b42-4e13-83c3-5d91a98eb7aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaa6d2-74d8499dd4-wfpmb" Oct 30 20:01:34.582129 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:34.582090 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvfq6\" (UniqueName: \"kubernetes.io/projected/510eee0c-6b42-4e13-83c3-5d91a98eb7aa-kube-api-access-nvfq6\") pod \"maas-api-e2e-ipp-a-eaa6d2-74d8499dd4-wfpmb\" (UID: \"510eee0c-6b42-4e13-83c3-5d91a98eb7aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaa6d2-74d8499dd4-wfpmb" Oct 30 20:01:34.582302 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:01:34.582146 4292 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaa6d2-serving-cert: secret "maas-api-e2e-ipp-a-eaa6d2-serving-cert" not found Oct 30 20:01:34.582302 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:01:34.582215 4292 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/510eee0c-6b42-4e13-83c3-5d91a98eb7aa-maas-api-tls podName:510eee0c-6b42-4e13-83c3-5d91a98eb7aa nodeName:}" failed. No retries permitted until 2026-10-30 20:01:35.082193504 +0000 UTC m=+2198.600508957 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/510eee0c-6b42-4e13-83c3-5d91a98eb7aa-maas-api-tls") pod "maas-api-e2e-ipp-a-eaa6d2-74d8499dd4-wfpmb" (UID: "510eee0c-6b42-4e13-83c3-5d91a98eb7aa") : secret "maas-api-e2e-ipp-a-eaa6d2-serving-cert" not found Oct 30 20:01:34.582683 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:34.582645 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/510eee0c-6b42-4e13-83c3-5d91a98eb7aa-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-eaa6d2-74d8499dd4-wfpmb\" (UID: \"510eee0c-6b42-4e13-83c3-5d91a98eb7aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaa6d2-74d8499dd4-wfpmb" Oct 30 20:01:34.584793 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:34.584767 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/510eee0c-6b42-4e13-83c3-5d91a98eb7aa-metrics-certs\") pod \"maas-api-e2e-ipp-a-eaa6d2-74d8499dd4-wfpmb\" (UID: \"510eee0c-6b42-4e13-83c3-5d91a98eb7aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaa6d2-74d8499dd4-wfpmb" Oct 30 20:01:34.647012 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:34.646978 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvfq6\" (UniqueName: \"kubernetes.io/projected/510eee0c-6b42-4e13-83c3-5d91a98eb7aa-kube-api-access-nvfq6\") pod \"maas-api-e2e-ipp-a-eaa6d2-74d8499dd4-wfpmb\" (UID: \"510eee0c-6b42-4e13-83c3-5d91a98eb7aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaa6d2-74d8499dd4-wfpmb" Oct 30 20:01:34.949336 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:34.949296 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr" Oct 30 20:01:34.949637 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:34.949549 4292 patch_prober.go:29] interesting pod/e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 20:01:34.949637 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:34.949603 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr" podUID="cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 20:01:35.088131 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:35.088099 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/510eee0c-6b42-4e13-83c3-5d91a98eb7aa-maas-api-tls\") pod \"maas-api-e2e-ipp-a-eaa6d2-74d8499dd4-wfpmb\" (UID: \"510eee0c-6b42-4e13-83c3-5d91a98eb7aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaa6d2-74d8499dd4-wfpmb" Oct 30 20:01:35.090655 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:35.090628 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/510eee0c-6b42-4e13-83c3-5d91a98eb7aa-maas-api-tls\") pod \"maas-api-e2e-ipp-a-eaa6d2-74d8499dd4-wfpmb\" (UID: \"510eee0c-6b42-4e13-83c3-5d91a98eb7aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaa6d2-74d8499dd4-wfpmb" Oct 30 20:01:35.291174 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:35.291141 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaa6d2-74d8499dd4-wfpmb" Oct 30 20:01:35.712736 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:35.712702 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaa6d2-74d8499dd4-wfpmb"] Oct 30 20:01:35.713919 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:01:35.713888 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod510eee0c_6b42_4e13_83c3_5d91a98eb7aa.slice/crio-bab5966bcc559344598cee19565e0838a6534dfeacf1a7974149288c3e6dfd35 WatchSource:0}: Error finding container bab5966bcc559344598cee19565e0838a6534dfeacf1a7974149288c3e6dfd35: Status 404 returned error can't find the container with id bab5966bcc559344598cee19565e0838a6534dfeacf1a7974149288c3e6dfd35 Oct 30 20:01:35.974866 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:35.974777 4292 patch_prober.go:29] interesting pod/e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 20:01:35.974866 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:35.974833 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr" podUID="cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 20:01:36.190484 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:36.190447 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaa6d2-74d8499dd4-wfpmb" event={"ID":"510eee0c-6b42-4e13-83c3-5d91a98eb7aa","Type":"ContainerStarted","Data":"bab5966bcc559344598cee19565e0838a6534dfeacf1a7974149288c3e6dfd35"} Oct 30 20:01:36.953443 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:36.949594 4292 patch_prober.go:29] interesting pod/e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 20:01:36.953443 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:36.949651 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr" podUID="cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 20:01:37.197524 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:37.197481 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaa6d2-74d8499dd4-wfpmb" event={"ID":"510eee0c-6b42-4e13-83c3-5d91a98eb7aa","Type":"ContainerStarted","Data":"f7aad4990e9baf0b6698cbcce55af25c2aff110536194d28891aa633caa12827"} Oct 30 20:01:37.197691 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:37.197552 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaa6d2-74d8499dd4-wfpmb" Oct 30 20:01:37.337625 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:37.337575 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaa6d2-74d8499dd4-wfpmb" podStartSLOduration=2.379248288 podStartE2EDuration="3.337557641s" totalImagesPullingTime="958.309353ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:37.335985981 +0000 UTC m=+2200.854301438" watchObservedRunningTime="2026-10-30 20:01:37.337557641 +0000 UTC m=+2200.855873160" Oct 30 20:01:37.963377 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:37.960758 4292 patch_prober.go:29] interesting pod/e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 20:01:37.963377 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:37.960834 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr" podUID="cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 20:01:38.970376 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:38.969464 4292 patch_prober.go:29] interesting pod/e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 20:01:38.970376 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:38.969550 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr" podUID="cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 20:01:39.972970 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:39.970802 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr" Oct 30 20:01:39.972970 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:39.972160 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr" Oct 30 20:01:39.973565 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:39.973416 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr" Oct 30 20:01:43.209076 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:01:43.209045 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaa6d2-74d8499dd4-wfpmb" Oct 30 20:01:43.221450 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:01:43.221419 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:02:13.221961 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:02:13.221915 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:02:34.545793 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:02:34.545761 4292 generic.go:356] "Generic (PLEG): container finished" podID="61b24a37-2664-40ff-a96e-58f2d44e1045" containerID="de7e331af7e302ee6ae20cd723fd13e3b790e95de0a254d4bba55c5c01a7b89b" exitCode=0 Oct 30 20:02:34.546270 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:02:34.545822 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9" event={"ID":"61b24a37-2664-40ff-a96e-58f2d44e1045","Type":"ContainerDied","Data":"de7e331af7e302ee6ae20cd723fd13e3b790e95de0a254d4bba55c5c01a7b89b"} Oct 30 20:02:34.546270 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:02:34.546213 4292 scope.go:122] "RemoveContainer" containerID="de7e331af7e302ee6ae20cd723fd13e3b790e95de0a254d4bba55c5c01a7b89b" Oct 30 20:02:34.547936 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:02:34.547903 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:02:34.548055 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:02:34.547991 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:02:34.548055 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:02:34.548041 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:02:35.553251 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:02:35.553207 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9" event={"ID":"61b24a37-2664-40ff-a96e-58f2d44e1045","Type":"ContainerStarted","Data":"9199cf36052feef81c8921548c3bf80f60d62b36c6f1e168967befd0bba1ebc7"} Oct 30 20:02:36.260053 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:02:36.260013 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9" Oct 30 20:02:36.260997 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:02:36.260971 4292 patch_prober.go:29] interesting pod/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 20:02:36.261123 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:02:36.261019 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9" podUID="61b24a37-2664-40ff-a96e-58f2d44e1045" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 20:02:36.809243 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:02:36.809214 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9" Oct 30 20:02:37.267437 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:02:37.259792 4292 patch_prober.go:29] interesting pod/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 20:02:37.267437 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:02:37.259851 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9" podUID="61b24a37-2664-40ff-a96e-58f2d44e1045" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 20:02:38.270376 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:02:38.264932 4292 patch_prober.go:29] interesting pod/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 20:02:38.270376 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:02:38.264991 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9" podUID="61b24a37-2664-40ff-a96e-58f2d44e1045" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 20:02:39.266776 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:02:39.266738 4292 patch_prober.go:29] interesting pod/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 20:02:39.267085 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:02:39.267056 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9" podUID="61b24a37-2664-40ff-a96e-58f2d44e1045" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 20:02:40.274765 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:02:40.274726 4292 patch_prober.go:29] interesting pod/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 20:02:40.275170 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:02:40.274785 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9" podUID="61b24a37-2664-40ff-a96e-58f2d44e1045" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 20:02:41.264740 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:02:41.264701 4292 patch_prober.go:29] interesting pod/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 20:02:41.265016 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:02:41.264988 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9" podUID="61b24a37-2664-40ff-a96e-58f2d44e1045" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 20:02:42.264233 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:02:42.264203 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9" Oct 30 20:02:42.265267 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:02:42.265246 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9" Oct 30 20:02:43.221301 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:02:43.221264 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:02:52.108155 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:02:52.108120 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3f9b9b-66ccc5955-mk88g"] Oct 30 20:02:55.127078 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:02:55.127048 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9"] Oct 30 20:02:55.127598 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:02:55.127269 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9" podUID="4f2b3150-1217-44bc-976a-65c86263223a" containerName="istio-proxy" containerID="cri-o://009626f5a7dc245ef9554b5c86581a5211c4fb43ca841e6dc21f5d2b4949e0ba" gracePeriod=30 Oct 30 20:02:57.108826 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:02:57.108770 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f9b9b-66ccc5955-mk88g" podUID="8fabac46-c08c-45bf-9cfe-b9b139aa4ebc" containerName="payload-pre-processing" containerID="cri-o://9fd95aa15059c17d7ef251dd5aacdda4dcee22c5a5822c86a7941bfc828f2df5" gracePeriod=25 Oct 30 20:02:57.364825 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:02:57.364760 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f9b9b-66ccc5955-mk88g" Oct 30 20:02:57.427619 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:02:57.427587 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fabac46-c08c-45bf-9cfe-b9b139aa4ebc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fabac46-c08c-45bf-9cfe-b9b139aa4ebc-plugins-config-volume\") pod \"8fabac46-c08c-45bf-9cfe-b9b139aa4ebc\" (UID: \"8fabac46-c08c-45bf-9cfe-b9b139aa4ebc\") " Oct 30 20:02:57.427789 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:02:57.427696 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fabac46-c08c-45bf-9cfe-b9b139aa4ebc-kube-api-access-hnkhm\" (UniqueName: \"kubernetes.io/projected/8fabac46-c08c-45bf-9cfe-b9b139aa4ebc-kube-api-access-hnkhm\") pod \"8fabac46-c08c-45bf-9cfe-b9b139aa4ebc\" (UID: \"8fabac46-c08c-45bf-9cfe-b9b139aa4ebc\") " Oct 30 20:02:57.427959 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:02:57.427936 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fabac46-c08c-45bf-9cfe-b9b139aa4ebc-plugins-config-volume" pod "8fabac46-c08c-45bf-9cfe-b9b139aa4ebc" (UID: "8fabac46-c08c-45bf-9cfe-b9b139aa4ebc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:57.429752 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:02:57.429727 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fabac46-c08c-45bf-9cfe-b9b139aa4ebc-kube-api-access-hnkhm" pod "8fabac46-c08c-45bf-9cfe-b9b139aa4ebc" (UID: "8fabac46-c08c-45bf-9cfe-b9b139aa4ebc"). InnerVolumeSpecName "kube-api-access-hnkhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:57.529161 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:02:57.529127 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnkhm\" (UniqueName: \"kubernetes.io/projected/8fabac46-c08c-45bf-9cfe-b9b139aa4ebc-kube-api-access-hnkhm\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:57.529161 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:02:57.529156 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fabac46-c08c-45bf-9cfe-b9b139aa4ebc-plugins-config-volume\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:57.803474 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:02:57.803429 4292 generic.go:356] "Generic (PLEG): container finished" podID="8fabac46-c08c-45bf-9cfe-b9b139aa4ebc" containerID="9fd95aa15059c17d7ef251dd5aacdda4dcee22c5a5822c86a7941bfc828f2df5" exitCode=0 Oct 30 20:02:57.803674 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:02:57.803479 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f9b9b-66ccc5955-mk88g" Oct 30 20:02:57.803674 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:02:57.803507 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f9b9b-66ccc5955-mk88g" event={"ID":"8fabac46-c08c-45bf-9cfe-b9b139aa4ebc","Type":"ContainerDied","Data":"9fd95aa15059c17d7ef251dd5aacdda4dcee22c5a5822c86a7941bfc828f2df5"} Oct 30 20:02:57.803674 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:02:57.803563 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f9b9b-66ccc5955-mk88g" event={"ID":"8fabac46-c08c-45bf-9cfe-b9b139aa4ebc","Type":"ContainerDied","Data":"a37eef9c655f70bd77511ce8af559d93af2434524ac7c9e2cadf6874899f049f"} Oct 30 20:02:57.803674 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:02:57.803585 4292 scope.go:122] "RemoveContainer" containerID="9fd95aa15059c17d7ef251dd5aacdda4dcee22c5a5822c86a7941bfc828f2df5" Oct 30 20:02:57.818754 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:02:57.818618 4292 scope.go:122] "RemoveContainer" containerID="9fd95aa15059c17d7ef251dd5aacdda4dcee22c5a5822c86a7941bfc828f2df5" Oct 30 20:02:57.819025 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:02:57.818997 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fd95aa15059c17d7ef251dd5aacdda4dcee22c5a5822c86a7941bfc828f2df5\": container with ID starting with 9fd95aa15059c17d7ef251dd5aacdda4dcee22c5a5822c86a7941bfc828f2df5 not found: ID does not exist" containerID="9fd95aa15059c17d7ef251dd5aacdda4dcee22c5a5822c86a7941bfc828f2df5" Oct 30 20:02:57.819119 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:02:57.819038 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fd95aa15059c17d7ef251dd5aacdda4dcee22c5a5822c86a7941bfc828f2df5"} err="failed to get container status \"9fd95aa15059c17d7ef251dd5aacdda4dcee22c5a5822c86a7941bfc828f2df5\": rpc error: code = NotFound desc = could not find container \"9fd95aa15059c17d7ef251dd5aacdda4dcee22c5a5822c86a7941bfc828f2df5\": container with ID starting with 9fd95aa15059c17d7ef251dd5aacdda4dcee22c5a5822c86a7941bfc828f2df5 not found: ID does not exist" Oct 30 20:02:57.989607 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:02:57.989567 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3f9b9b-66ccc5955-mk88g"] Oct 30 20:02:58.040196 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:02:58.040167 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3f9b9b-66ccc5955-mk88g"] Oct 30 20:02:58.582528 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:02:58.582494 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fabac46-c08c-45bf-9cfe-b9b139aa4ebc" path="/var/lib/kubelet/pods/8fabac46-c08c-45bf-9cfe-b9b139aa4ebc/volumes" Oct 30 20:03:00.391149 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:00.391093 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9" Oct 30 20:03:00.467327 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:00.467297 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f2b3150-1217-44bc-976a-65c86263223a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f2b3150-1217-44bc-976a-65c86263223a-istio-data\") pod \"4f2b3150-1217-44bc-976a-65c86263223a\" (UID: \"4f2b3150-1217-44bc-976a-65c86263223a\") " Oct 30 20:03:00.467485 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:00.467412 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f2b3150-1217-44bc-976a-65c86263223a-kube-api-access-hxfb9\" (UniqueName: \"kubernetes.io/projected/4f2b3150-1217-44bc-976a-65c86263223a-kube-api-access-hxfb9\") pod \"4f2b3150-1217-44bc-976a-65c86263223a\" (UID: \"4f2b3150-1217-44bc-976a-65c86263223a\") " Oct 30 20:03:00.467485 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:00.467446 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4f2b3150-1217-44bc-976a-65c86263223a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f2b3150-1217-44bc-976a-65c86263223a-istio-podinfo\") pod \"4f2b3150-1217-44bc-976a-65c86263223a\" (UID: \"4f2b3150-1217-44bc-976a-65c86263223a\") " Oct 30 20:03:00.467594 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:00.467522 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f2b3150-1217-44bc-976a-65c86263223a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f2b3150-1217-44bc-976a-65c86263223a-workload-certs\") pod \"4f2b3150-1217-44bc-976a-65c86263223a\" (UID: \"4f2b3150-1217-44bc-976a-65c86263223a\") " Oct 30 20:03:00.467594 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:00.467560 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f2b3150-1217-44bc-976a-65c86263223a-istio-data" pod "4f2b3150-1217-44bc-976a-65c86263223a" (UID: "4f2b3150-1217-44bc-976a-65c86263223a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:00.467594 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:00.467580 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f2b3150-1217-44bc-976a-65c86263223a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f2b3150-1217-44bc-976a-65c86263223a-workload-socket\") pod \"4f2b3150-1217-44bc-976a-65c86263223a\" (UID: \"4f2b3150-1217-44bc-976a-65c86263223a\") " Oct 30 20:03:00.467754 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:00.467657 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f2b3150-1217-44bc-976a-65c86263223a-istio-token\" (UniqueName: \"kubernetes.io/projected/4f2b3150-1217-44bc-976a-65c86263223a-istio-token\") pod \"4f2b3150-1217-44bc-976a-65c86263223a\" (UID: \"4f2b3150-1217-44bc-976a-65c86263223a\") " Oct 30 20:03:00.467754 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:00.467712 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f2b3150-1217-44bc-976a-65c86263223a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f2b3150-1217-44bc-976a-65c86263223a-istio-envoy\") pod \"4f2b3150-1217-44bc-976a-65c86263223a\" (UID: \"4f2b3150-1217-44bc-976a-65c86263223a\") " Oct 30 20:03:00.467864 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:00.467750 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f2b3150-1217-44bc-976a-65c86263223a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f2b3150-1217-44bc-976a-65c86263223a-credential-socket\") pod \"4f2b3150-1217-44bc-976a-65c86263223a\" (UID: \"4f2b3150-1217-44bc-976a-65c86263223a\") " Oct 30 20:03:00.467864 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:00.467779 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f2b3150-1217-44bc-976a-65c86263223a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f2b3150-1217-44bc-976a-65c86263223a-istiod-ca-cert\") pod \"4f2b3150-1217-44bc-976a-65c86263223a\" (UID: \"4f2b3150-1217-44bc-976a-65c86263223a\") " Oct 30 20:03:00.467864 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:00.467832 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f2b3150-1217-44bc-976a-65c86263223a-workload-certs" pod "4f2b3150-1217-44bc-976a-65c86263223a" (UID: "4f2b3150-1217-44bc-976a-65c86263223a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:00.467864 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:00.467847 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f2b3150-1217-44bc-976a-65c86263223a-workload-socket" pod "4f2b3150-1217-44bc-976a-65c86263223a" (UID: "4f2b3150-1217-44bc-976a-65c86263223a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:00.468083 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:00.468036 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f2b3150-1217-44bc-976a-65c86263223a-credential-socket" pod "4f2b3150-1217-44bc-976a-65c86263223a" (UID: "4f2b3150-1217-44bc-976a-65c86263223a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:00.468217 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:00.468197 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f2b3150-1217-44bc-976a-65c86263223a-istiod-ca-cert" pod "4f2b3150-1217-44bc-976a-65c86263223a" (UID: "4f2b3150-1217-44bc-976a-65c86263223a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:03:00.468308 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:00.468289 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f2b3150-1217-44bc-976a-65c86263223a-workload-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:00.468401 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:00.468313 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f2b3150-1217-44bc-976a-65c86263223a-workload-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:00.468401 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:00.468329 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f2b3150-1217-44bc-976a-65c86263223a-credential-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:00.468401 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:00.468342 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f2b3150-1217-44bc-976a-65c86263223a-istiod-ca-cert\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:00.468401 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:00.468375 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f2b3150-1217-44bc-976a-65c86263223a-istio-data\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:00.469755 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:00.469732 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f2b3150-1217-44bc-976a-65c86263223a-istio-podinfo" pod "4f2b3150-1217-44bc-976a-65c86263223a" (UID: "4f2b3150-1217-44bc-976a-65c86263223a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:03:00.469854 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:00.469753 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f2b3150-1217-44bc-976a-65c86263223a-kube-api-access-hxfb9" pod "4f2b3150-1217-44bc-976a-65c86263223a" (UID: "4f2b3150-1217-44bc-976a-65c86263223a"). InnerVolumeSpecName "kube-api-access-hxfb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:00.470084 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:00.470070 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f2b3150-1217-44bc-976a-65c86263223a-istio-token" pod "4f2b3150-1217-44bc-976a-65c86263223a" (UID: "4f2b3150-1217-44bc-976a-65c86263223a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:00.470155 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:00.470141 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f2b3150-1217-44bc-976a-65c86263223a-istio-envoy" pod "4f2b3150-1217-44bc-976a-65c86263223a" (UID: "4f2b3150-1217-44bc-976a-65c86263223a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:00.569463 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:00.569427 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f2b3150-1217-44bc-976a-65c86263223a-istio-token\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:00.569463 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:00.569452 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f2b3150-1217-44bc-976a-65c86263223a-istio-envoy\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:00.569463 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:00.569463 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxfb9\" (UniqueName: \"kubernetes.io/projected/4f2b3150-1217-44bc-976a-65c86263223a-kube-api-access-hxfb9\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:00.569463 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:00.569476 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f2b3150-1217-44bc-976a-65c86263223a-istio-podinfo\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:00.831499 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:00.831468 4292 generic.go:356] "Generic (PLEG): container finished" podID="4f2b3150-1217-44bc-976a-65c86263223a" containerID="009626f5a7dc245ef9554b5c86581a5211c4fb43ca841e6dc21f5d2b4949e0ba" exitCode=0 Oct 30 20:03:00.831675 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:00.831518 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9" Oct 30 20:03:00.831675 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:00.831554 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9" event={"ID":"4f2b3150-1217-44bc-976a-65c86263223a","Type":"ContainerDied","Data":"009626f5a7dc245ef9554b5c86581a5211c4fb43ca841e6dc21f5d2b4949e0ba"} Oct 30 20:03:00.831675 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:00.831608 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9" event={"ID":"4f2b3150-1217-44bc-976a-65c86263223a","Type":"ContainerDied","Data":"d6154a0575fdc6ed4f1e178272df401c0f0c5dccd746ca68664b0c41972a2b49"} Oct 30 20:03:00.831675 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:00.831630 4292 scope.go:122] "RemoveContainer" containerID="009626f5a7dc245ef9554b5c86581a5211c4fb43ca841e6dc21f5d2b4949e0ba" Oct 30 20:03:00.840830 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:00.840810 4292 scope.go:122] "RemoveContainer" containerID="009626f5a7dc245ef9554b5c86581a5211c4fb43ca841e6dc21f5d2b4949e0ba" Oct 30 20:03:00.841080 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:03:00.841060 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"009626f5a7dc245ef9554b5c86581a5211c4fb43ca841e6dc21f5d2b4949e0ba\": container with ID starting with 009626f5a7dc245ef9554b5c86581a5211c4fb43ca841e6dc21f5d2b4949e0ba not found: ID does not exist" containerID="009626f5a7dc245ef9554b5c86581a5211c4fb43ca841e6dc21f5d2b4949e0ba" Oct 30 20:03:00.841154 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:00.841090 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"009626f5a7dc245ef9554b5c86581a5211c4fb43ca841e6dc21f5d2b4949e0ba"} err="failed to get container status \"009626f5a7dc245ef9554b5c86581a5211c4fb43ca841e6dc21f5d2b4949e0ba\": rpc error: code = NotFound desc = could not find container \"009626f5a7dc245ef9554b5c86581a5211c4fb43ca841e6dc21f5d2b4949e0ba\": container with ID starting with 009626f5a7dc245ef9554b5c86581a5211c4fb43ca841e6dc21f5d2b4949e0ba not found: ID does not exist" Oct 30 20:03:01.004941 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:01.004910 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9"] Oct 30 20:03:01.055694 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:01.055666 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-bwqd9"] Oct 30 20:03:01.993801 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:01.993769 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h"] Oct 30 20:03:01.994434 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:01.994259 4292 cpu_manager.go:414] "containerMap: removing container" podUID="8fabac46-c08c-45bf-9cfe-b9b139aa4ebc" containerName="payload-pre-processing" Oct 30 20:03:01.994434 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:01.994276 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fabac46-c08c-45bf-9cfe-b9b139aa4ebc" containerName="payload-pre-processing" Oct 30 20:03:01.994434 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:01.994295 4292 cpu_manager.go:414] "containerMap: removing container" podUID="4f2b3150-1217-44bc-976a-65c86263223a" containerName="istio-proxy" Oct 30 20:03:01.994434 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:01.994301 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f2b3150-1217-44bc-976a-65c86263223a" containerName="istio-proxy" Oct 30 20:03:01.994707 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:01.994442 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fabac46-c08c-45bf-9cfe-b9b139aa4ebc" containerName="payload-pre-processing" Oct 30 20:03:01.994707 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:01.994456 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f2b3150-1217-44bc-976a-65c86263223a" containerName="istio-proxy" Oct 30 20:03:01.998263 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:01.998244 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h" Oct 30 20:03:02.066709 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:02.066673 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-c60685-openshift-default-dockercfg-bf72f\"" Oct 30 20:03:02.131782 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:02.130175 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h"] Oct 30 20:03:02.205381 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:02.203269 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/607ab3e3-4a07-41ef-b1d9-983ddc238675-workload-certs\") pod \"e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h\" (UID: \"607ab3e3-4a07-41ef-b1d9-983ddc238675\") " pod="openshift-ingress/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h" Oct 30 20:03:02.205381 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:02.203333 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc28h\" (UniqueName: \"kubernetes.io/projected/607ab3e3-4a07-41ef-b1d9-983ddc238675-kube-api-access-bc28h\") pod \"e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h\" (UID: \"607ab3e3-4a07-41ef-b1d9-983ddc238675\") " pod="openshift-ingress/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h" Oct 30 20:03:02.205381 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:02.203452 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/607ab3e3-4a07-41ef-b1d9-983ddc238675-istio-envoy\") pod \"e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h\" (UID: \"607ab3e3-4a07-41ef-b1d9-983ddc238675\") " pod="openshift-ingress/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h" Oct 30 20:03:02.205381 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:02.203485 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/607ab3e3-4a07-41ef-b1d9-983ddc238675-istio-data\") pod \"e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h\" (UID: \"607ab3e3-4a07-41ef-b1d9-983ddc238675\") " pod="openshift-ingress/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h" Oct 30 20:03:02.205381 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:02.203548 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/607ab3e3-4a07-41ef-b1d9-983ddc238675-workload-socket\") pod \"e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h\" (UID: \"607ab3e3-4a07-41ef-b1d9-983ddc238675\") " pod="openshift-ingress/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h" Oct 30 20:03:02.205381 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:02.203621 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/607ab3e3-4a07-41ef-b1d9-983ddc238675-credential-socket\") pod \"e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h\" (UID: \"607ab3e3-4a07-41ef-b1d9-983ddc238675\") " pod="openshift-ingress/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h" Oct 30 20:03:02.205381 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:02.203667 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/607ab3e3-4a07-41ef-b1d9-983ddc238675-istio-token\") pod \"e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h\" (UID: \"607ab3e3-4a07-41ef-b1d9-983ddc238675\") " pod="openshift-ingress/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h" Oct 30 20:03:02.205381 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:02.203695 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/607ab3e3-4a07-41ef-b1d9-983ddc238675-istio-podinfo\") pod \"e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h\" (UID: \"607ab3e3-4a07-41ef-b1d9-983ddc238675\") " pod="openshift-ingress/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h" Oct 30 20:03:02.205381 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:02.203739 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/607ab3e3-4a07-41ef-b1d9-983ddc238675-istiod-ca-cert\") pod \"e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h\" (UID: \"607ab3e3-4a07-41ef-b1d9-983ddc238675\") " pod="openshift-ingress/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h" Oct 30 20:03:02.304778 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:02.304743 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/607ab3e3-4a07-41ef-b1d9-983ddc238675-credential-socket\") pod \"e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h\" (UID: \"607ab3e3-4a07-41ef-b1d9-983ddc238675\") " pod="openshift-ingress/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h" Oct 30 20:03:02.305005 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:02.304792 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/607ab3e3-4a07-41ef-b1d9-983ddc238675-istio-token\") pod \"e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h\" (UID: \"607ab3e3-4a07-41ef-b1d9-983ddc238675\") " pod="openshift-ingress/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h" Oct 30 20:03:02.305005 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:02.304815 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/607ab3e3-4a07-41ef-b1d9-983ddc238675-istio-podinfo\") pod \"e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h\" (UID: \"607ab3e3-4a07-41ef-b1d9-983ddc238675\") " pod="openshift-ingress/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h" Oct 30 20:03:02.305005 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:02.304851 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/607ab3e3-4a07-41ef-b1d9-983ddc238675-istiod-ca-cert\") pod \"e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h\" (UID: \"607ab3e3-4a07-41ef-b1d9-983ddc238675\") " pod="openshift-ingress/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h" Oct 30 20:03:02.305183 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:02.305024 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/607ab3e3-4a07-41ef-b1d9-983ddc238675-workload-certs\") pod \"e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h\" (UID: \"607ab3e3-4a07-41ef-b1d9-983ddc238675\") " pod="openshift-ingress/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h" Oct 30 20:03:02.305183 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:02.305081 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc28h\" (UniqueName: \"kubernetes.io/projected/607ab3e3-4a07-41ef-b1d9-983ddc238675-kube-api-access-bc28h\") pod \"e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h\" (UID: \"607ab3e3-4a07-41ef-b1d9-983ddc238675\") " pod="openshift-ingress/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h" Oct 30 20:03:02.305299 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:02.305200 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/607ab3e3-4a07-41ef-b1d9-983ddc238675-istio-envoy\") pod \"e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h\" (UID: \"607ab3e3-4a07-41ef-b1d9-983ddc238675\") " pod="openshift-ingress/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h" Oct 30 20:03:02.305299 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:02.305230 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/607ab3e3-4a07-41ef-b1d9-983ddc238675-istio-data\") pod \"e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h\" (UID: \"607ab3e3-4a07-41ef-b1d9-983ddc238675\") " pod="openshift-ingress/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h" Oct 30 20:03:02.305299 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:02.305242 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/607ab3e3-4a07-41ef-b1d9-983ddc238675-credential-socket\") pod \"e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h\" (UID: \"607ab3e3-4a07-41ef-b1d9-983ddc238675\") " pod="openshift-ingress/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h" Oct 30 20:03:02.305543 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:02.305330 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/607ab3e3-4a07-41ef-b1d9-983ddc238675-workload-socket\") pod \"e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h\" (UID: \"607ab3e3-4a07-41ef-b1d9-983ddc238675\") " pod="openshift-ingress/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h" Oct 30 20:03:02.305543 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:02.305471 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/607ab3e3-4a07-41ef-b1d9-983ddc238675-workload-certs\") pod \"e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h\" (UID: \"607ab3e3-4a07-41ef-b1d9-983ddc238675\") " pod="openshift-ingress/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h" Oct 30 20:03:02.305660 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:02.305619 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/607ab3e3-4a07-41ef-b1d9-983ddc238675-istio-data\") pod \"e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h\" (UID: \"607ab3e3-4a07-41ef-b1d9-983ddc238675\") " pod="openshift-ingress/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h" Oct 30 20:03:02.305745 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:02.305713 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/607ab3e3-4a07-41ef-b1d9-983ddc238675-istiod-ca-cert\") pod \"e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h\" (UID: \"607ab3e3-4a07-41ef-b1d9-983ddc238675\") " pod="openshift-ingress/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h" Oct 30 20:03:02.305745 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:02.305733 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/607ab3e3-4a07-41ef-b1d9-983ddc238675-workload-socket\") pod \"e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h\" (UID: \"607ab3e3-4a07-41ef-b1d9-983ddc238675\") " pod="openshift-ingress/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h" Oct 30 20:03:02.307666 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:02.307645 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/607ab3e3-4a07-41ef-b1d9-983ddc238675-istio-envoy\") pod \"e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h\" (UID: \"607ab3e3-4a07-41ef-b1d9-983ddc238675\") " pod="openshift-ingress/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h" Oct 30 20:03:02.307925 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:02.307904 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/607ab3e3-4a07-41ef-b1d9-983ddc238675-istio-podinfo\") pod \"e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h\" (UID: \"607ab3e3-4a07-41ef-b1d9-983ddc238675\") " pod="openshift-ingress/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h" Oct 30 20:03:02.367613 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:02.367576 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/607ab3e3-4a07-41ef-b1d9-983ddc238675-istio-token\") pod \"e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h\" (UID: \"607ab3e3-4a07-41ef-b1d9-983ddc238675\") " pod="openshift-ingress/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h" Oct 30 20:03:02.373571 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:02.373540 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc28h\" (UniqueName: \"kubernetes.io/projected/607ab3e3-4a07-41ef-b1d9-983ddc238675-kube-api-access-bc28h\") pod \"e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h\" (UID: \"607ab3e3-4a07-41ef-b1d9-983ddc238675\") " pod="openshift-ingress/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h" Oct 30 20:03:02.584217 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:02.584137 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f2b3150-1217-44bc-976a-65c86263223a" path="/var/lib/kubelet/pods/4f2b3150-1217-44bc-976a-65c86263223a/volumes" Oct 30 20:03:02.631292 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:02.631257 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h" Oct 30 20:03:03.085121 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:03.085087 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h"] Oct 30 20:03:03.086285 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:03:03.086257 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod607ab3e3_4a07_41ef_b1d9_983ddc238675.slice/crio-c9b644e11c33545163a21af537811a24df1df026e5b2b5678c169a21f0736b43 WatchSource:0}: Error finding container c9b644e11c33545163a21af537811a24df1df026e5b2b5678c169a21f0736b43: Status 404 returned error can't find the container with id c9b644e11c33545163a21af537811a24df1df026e5b2b5678c169a21f0736b43 Oct 30 20:03:03.088850 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:03.088821 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:03:03.088926 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:03.088886 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:03:03.088926 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:03.088914 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:03:03.852185 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:03.852151 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h" event={"ID":"607ab3e3-4a07-41ef-b1d9-983ddc238675","Type":"ContainerStarted","Data":"de5d656416bcad3602c679675ec8e66f99ba27958850ecdb72f7fe4c58a312bd"} Oct 30 20:03:03.852397 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:03.852202 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h" event={"ID":"607ab3e3-4a07-41ef-b1d9-983ddc238675","Type":"ContainerStarted","Data":"c9b644e11c33545163a21af537811a24df1df026e5b2b5678c169a21f0736b43"} Oct 30 20:03:04.081271 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:04.081217 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h" podStartSLOduration=3.081200979 podStartE2EDuration="3.081200979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:04.028640712 +0000 UTC m=+2287.546956189" watchObservedRunningTime="2026-10-30 20:03:04.081200979 +0000 UTC m=+2287.599516435" Oct 30 20:03:04.632093 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:04.632062 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h" Oct 30 20:03:04.637830 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:04.637806 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h" Oct 30 20:03:04.858018 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:04.857984 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h" Oct 30 20:03:04.858888 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:04.858865 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h" Oct 30 20:03:11.802897 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:11.802816 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c60685-5d446879b7-d72xw"] Oct 30 20:03:11.805829 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:11.805810 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c60685-5d446879b7-d72xw" Oct 30 20:03:11.861927 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:11.861898 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c60685-5d446879b7-d72xw"] Oct 30 20:03:11.919693 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:11.919658 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-c60685\"" Oct 30 20:03:11.920083 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:11.920062 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-c60685-dockercfg-dsb8d\"" Oct 30 20:03:12.024032 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:12.022187 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxdcm\" (UniqueName: \"kubernetes.io/projected/9ec677ce-8380-4793-b6cf-02a092f9ffa5-kube-api-access-kxdcm\") pod \"payload-pre-processing-e2e-ipp-b-c60685-5d446879b7-d72xw\" (UID: \"9ec677ce-8380-4793-b6cf-02a092f9ffa5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c60685-5d446879b7-d72xw" Oct 30 20:03:12.024032 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:12.022262 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ec677ce-8380-4793-b6cf-02a092f9ffa5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c60685-5d446879b7-d72xw\" (UID: \"9ec677ce-8380-4793-b6cf-02a092f9ffa5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c60685-5d446879b7-d72xw" Oct 30 20:03:12.123952 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:12.123870 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxdcm\" (UniqueName: \"kubernetes.io/projected/9ec677ce-8380-4793-b6cf-02a092f9ffa5-kube-api-access-kxdcm\") pod \"payload-pre-processing-e2e-ipp-b-c60685-5d446879b7-d72xw\" (UID: \"9ec677ce-8380-4793-b6cf-02a092f9ffa5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c60685-5d446879b7-d72xw" Oct 30 20:03:12.123952 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:12.123918 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ec677ce-8380-4793-b6cf-02a092f9ffa5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c60685-5d446879b7-d72xw\" (UID: \"9ec677ce-8380-4793-b6cf-02a092f9ffa5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c60685-5d446879b7-d72xw" Oct 30 20:03:12.124701 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:12.124676 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ec677ce-8380-4793-b6cf-02a092f9ffa5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c60685-5d446879b7-d72xw\" (UID: \"9ec677ce-8380-4793-b6cf-02a092f9ffa5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c60685-5d446879b7-d72xw" Oct 30 20:03:12.186461 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:12.186429 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxdcm\" (UniqueName: \"kubernetes.io/projected/9ec677ce-8380-4793-b6cf-02a092f9ffa5-kube-api-access-kxdcm\") pod \"payload-pre-processing-e2e-ipp-b-c60685-5d446879b7-d72xw\" (UID: \"9ec677ce-8380-4793-b6cf-02a092f9ffa5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c60685-5d446879b7-d72xw" Oct 30 20:03:12.418383 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:12.418279 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c60685-5d446879b7-d72xw" Oct 30 20:03:12.795387 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:12.792289 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c60685-5d446879b7-d72xw"] Oct 30 20:03:12.801441 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:03:12.801414 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ec677ce_8380_4793_b6cf_02a092f9ffa5.slice/crio-5d30ae0b6f181be062b59ff43560eff78c50b8a577c0ef226c3e3d79c37900fa WatchSource:0}: Error finding container 5d30ae0b6f181be062b59ff43560eff78c50b8a577c0ef226c3e3d79c37900fa: Status 404 returned error can't find the container with id 5d30ae0b6f181be062b59ff43560eff78c50b8a577c0ef226c3e3d79c37900fa Oct 30 20:03:12.982487 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:12.982453 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c60685-5d446879b7-d72xw" event={"ID":"9ec677ce-8380-4793-b6cf-02a092f9ffa5","Type":"ContainerStarted","Data":"de2ddfb345a683bea6ff906963431b1d7521d76ab825f20176d2de0d6abd6626"} Oct 30 20:03:12.982942 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:12.982505 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c60685-5d446879b7-d72xw" event={"ID":"9ec677ce-8380-4793-b6cf-02a092f9ffa5","Type":"ContainerStarted","Data":"5d30ae0b6f181be062b59ff43560eff78c50b8a577c0ef226c3e3d79c37900fa"} Oct 30 20:03:12.982942 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:12.982552 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c60685-5d446879b7-d72xw" Oct 30 20:03:13.105507 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:13.105400 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c60685-5d446879b7-d72xw" podStartSLOduration=2.105385379 podStartE2EDuration="2.105385379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:13.103506084 +0000 UTC m=+2296.621821542" watchObservedRunningTime="2026-10-30 20:03:13.105385379 +0000 UTC m=+2296.623700836" Oct 30 20:03:13.242630 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:03:13.241736 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:03:23.991571 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:23.991538 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c60685-5d446879b7-d72xw" Oct 30 20:03:27.678484 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:27.678446 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f700fe-56cffff755-ck8pz"] Oct 30 20:03:27.681111 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:27.681091 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f700fe-56cffff755-ck8pz" Oct 30 20:03:27.737074 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:27.737040 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f700fe-56cffff755-ck8pz"] Oct 30 20:03:27.764193 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:27.764154 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slssp\" (UniqueName: \"kubernetes.io/projected/26b6bea7-3ad9-4c8d-85b9-fb2edca9afce-kube-api-access-slssp\") pod \"payload-pre-processing-e2e-ipp-b-f700fe-56cffff755-ck8pz\" (UID: \"26b6bea7-3ad9-4c8d-85b9-fb2edca9afce\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f700fe-56cffff755-ck8pz" Oct 30 20:03:27.764451 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:27.764430 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26b6bea7-3ad9-4c8d-85b9-fb2edca9afce-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f700fe-56cffff755-ck8pz\" (UID: \"26b6bea7-3ad9-4c8d-85b9-fb2edca9afce\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f700fe-56cffff755-ck8pz" Oct 30 20:03:27.784626 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:27.784596 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-f700fe\"" Oct 30 20:03:27.784793 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:27.784680 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-f700fe-dockercfg-6c5hf\"" Oct 30 20:03:27.865420 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:27.865384 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26b6bea7-3ad9-4c8d-85b9-fb2edca9afce-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f700fe-56cffff755-ck8pz\" (UID: \"26b6bea7-3ad9-4c8d-85b9-fb2edca9afce\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f700fe-56cffff755-ck8pz" Oct 30 20:03:27.865581 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:27.865551 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slssp\" (UniqueName: \"kubernetes.io/projected/26b6bea7-3ad9-4c8d-85b9-fb2edca9afce-kube-api-access-slssp\") pod \"payload-pre-processing-e2e-ipp-b-f700fe-56cffff755-ck8pz\" (UID: \"26b6bea7-3ad9-4c8d-85b9-fb2edca9afce\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f700fe-56cffff755-ck8pz" Oct 30 20:03:27.866174 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:27.866155 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26b6bea7-3ad9-4c8d-85b9-fb2edca9afce-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f700fe-56cffff755-ck8pz\" (UID: \"26b6bea7-3ad9-4c8d-85b9-fb2edca9afce\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f700fe-56cffff755-ck8pz" Oct 30 20:03:27.928084 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:27.928055 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slssp\" (UniqueName: \"kubernetes.io/projected/26b6bea7-3ad9-4c8d-85b9-fb2edca9afce-kube-api-access-slssp\") pod \"payload-pre-processing-e2e-ipp-b-f700fe-56cffff755-ck8pz\" (UID: \"26b6bea7-3ad9-4c8d-85b9-fb2edca9afce\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f700fe-56cffff755-ck8pz" Oct 30 20:03:27.993280 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:27.993211 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f700fe-56cffff755-ck8pz" Oct 30 20:03:28.370274 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:28.370238 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f700fe-56cffff755-ck8pz"] Oct 30 20:03:28.371326 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:03:28.371297 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26b6bea7_3ad9_4c8d_85b9_fb2edca9afce.slice/crio-b89d017f16acfe9732713230d2571187227eb1bb288daa4a73b4c4e7649fe615 WatchSource:0}: Error finding container b89d017f16acfe9732713230d2571187227eb1bb288daa4a73b4c4e7649fe615: Status 404 returned error can't find the container with id b89d017f16acfe9732713230d2571187227eb1bb288daa4a73b4c4e7649fe615 Oct 30 20:03:29.084338 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:29.084297 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f700fe-56cffff755-ck8pz" event={"ID":"26b6bea7-3ad9-4c8d-85b9-fb2edca9afce","Type":"ContainerStarted","Data":"5bb4cc923f663d15def9c3bf751f03d8a903f9b9ea59660ae1c98be69bd4b984"} Oct 30 20:03:29.084750 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:29.084395 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f700fe-56cffff755-ck8pz" event={"ID":"26b6bea7-3ad9-4c8d-85b9-fb2edca9afce","Type":"ContainerStarted","Data":"b89d017f16acfe9732713230d2571187227eb1bb288daa4a73b4c4e7649fe615"} Oct 30 20:03:29.084750 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:29.084495 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f700fe-56cffff755-ck8pz" Oct 30 20:03:29.204253 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:29.204203 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f700fe-56cffff755-ck8pz" podStartSLOduration=2.204190455 podStartE2EDuration="2.204190455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:29.203715149 +0000 UTC m=+2312.722030606" watchObservedRunningTime="2026-10-30 20:03:29.204190455 +0000 UTC m=+2312.722505912" Oct 30 20:03:40.093056 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:40.093026 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f700fe-56cffff755-ck8pz" Oct 30 20:03:43.220959 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:03:43.220920 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:03:48.752463 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:48.752431 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-77a92c-64d9fc577b-pbdb5"] Oct 30 20:03:48.869649 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:48.869616 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-77a92c-65f6f789f-rxgqj"] Oct 30 20:03:50.276373 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:50.275331 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ffe14b-2vrw7"] Oct 30 20:03:50.285871 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:50.285838 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ffe14b-2vrw7" Oct 30 20:03:50.329519 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:50.329490 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ffe14b-2vrw7"] Oct 30 20:03:50.341880 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:50.341854 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:03:50.342109 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:50.342090 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\"" Oct 30 20:03:50.354683 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:50.354657 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5246\" (UniqueName: \"kubernetes.io/projected/e49f4c77-14f4-4180-8929-5a40d7bbafba-kube-api-access-x5246\") pod \"maas-api-revoke-keys-e2e-ipp-a-ffe14b-2vrw7\" (UID: \"e49f4c77-14f4-4180-8929-5a40d7bbafba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ffe14b-2vrw7" Oct 30 20:03:50.354793 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:50.354713 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e49f4c77-14f4-4180-8929-5a40d7bbafba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ffe14b-2vrw7\" (UID: \"e49f4c77-14f4-4180-8929-5a40d7bbafba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ffe14b-2vrw7" Oct 30 20:03:50.354959 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:50.354942 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e49f4c77-14f4-4180-8929-5a40d7bbafba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ffe14b-2vrw7\" (UID: \"e49f4c77-14f4-4180-8929-5a40d7bbafba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ffe14b-2vrw7" Oct 30 20:03:50.464258 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:50.458214 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e49f4c77-14f4-4180-8929-5a40d7bbafba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ffe14b-2vrw7\" (UID: \"e49f4c77-14f4-4180-8929-5a40d7bbafba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ffe14b-2vrw7" Oct 30 20:03:50.464258 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:50.458453 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e49f4c77-14f4-4180-8929-5a40d7bbafba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ffe14b-2vrw7\" (UID: \"e49f4c77-14f4-4180-8929-5a40d7bbafba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ffe14b-2vrw7" Oct 30 20:03:50.464258 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:50.458523 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5246\" (UniqueName: \"kubernetes.io/projected/e49f4c77-14f4-4180-8929-5a40d7bbafba-kube-api-access-x5246\") pod \"maas-api-revoke-keys-e2e-ipp-a-ffe14b-2vrw7\" (UID: \"e49f4c77-14f4-4180-8929-5a40d7bbafba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ffe14b-2vrw7" Oct 30 20:03:50.464258 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:50.459660 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e49f4c77-14f4-4180-8929-5a40d7bbafba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ffe14b-2vrw7\" (UID: \"e49f4c77-14f4-4180-8929-5a40d7bbafba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ffe14b-2vrw7" Oct 30 20:03:50.478677 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:50.478644 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e49f4c77-14f4-4180-8929-5a40d7bbafba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ffe14b-2vrw7\" (UID: \"e49f4c77-14f4-4180-8929-5a40d7bbafba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ffe14b-2vrw7" Oct 30 20:03:50.521231 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:50.521199 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5246\" (UniqueName: \"kubernetes.io/projected/e49f4c77-14f4-4180-8929-5a40d7bbafba-kube-api-access-x5246\") pod \"maas-api-revoke-keys-e2e-ipp-a-ffe14b-2vrw7\" (UID: \"e49f4c77-14f4-4180-8929-5a40d7bbafba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ffe14b-2vrw7" Oct 30 20:03:50.623521 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:50.623434 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ffe14b-2vrw7" Oct 30 20:03:51.010935 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:51.010909 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ffe14b-2vrw7"] Oct 30 20:03:51.011813 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:03:51.011788 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode49f4c77_14f4_4180_8929_5a40d7bbafba.slice/crio-a5aee78ceb778ac8b610c83eeaa984004406911a80a8cca399541fd6181b4541 WatchSource:0}: Error finding container a5aee78ceb778ac8b610c83eeaa984004406911a80a8cca399541fd6181b4541: Status 404 returned error can't find the container with id a5aee78ceb778ac8b610c83eeaa984004406911a80a8cca399541fd6181b4541 Oct 30 20:03:51.268105 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:51.268077 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ffe14b-2vrw7" event={"ID":"e49f4c77-14f4-4180-8929-5a40d7bbafba","Type":"ContainerStarted","Data":"a5aee78ceb778ac8b610c83eeaa984004406911a80a8cca399541fd6181b4541"} Oct 30 20:03:52.308199 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:52.307088 4292 generic.go:356] "Generic (PLEG): container finished" podID="e49f4c77-14f4-4180-8929-5a40d7bbafba" containerID="1940e2aeddd2e903771b23ff253ecd1b222c1d581a35454abd3513f2966ed81c" exitCode=0 Oct 30 20:03:52.308199 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:52.307444 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ffe14b-2vrw7" event={"ID":"e49f4c77-14f4-4180-8929-5a40d7bbafba","Type":"ContainerDied","Data":"1940e2aeddd2e903771b23ff253ecd1b222c1d581a35454abd3513f2966ed81c"} Oct 30 20:03:53.462170 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:53.462143 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ffe14b-2vrw7" Oct 30 20:03:53.614074 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:53.613985 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e49f4c77-14f4-4180-8929-5a40d7bbafba-kube-api-access-x5246\" (UniqueName: \"kubernetes.io/projected/e49f4c77-14f4-4180-8929-5a40d7bbafba-kube-api-access-x5246\") pod \"e49f4c77-14f4-4180-8929-5a40d7bbafba\" (UID: \"e49f4c77-14f4-4180-8929-5a40d7bbafba\") " Oct 30 20:03:53.614225 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:53.614154 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e49f4c77-14f4-4180-8929-5a40d7bbafba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e49f4c77-14f4-4180-8929-5a40d7bbafba-maas-api-xks-ca\") pod \"e49f4c77-14f4-4180-8929-5a40d7bbafba\" (UID: \"e49f4c77-14f4-4180-8929-5a40d7bbafba\") " Oct 30 20:03:53.614262 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:53.614250 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e49f4c77-14f4-4180-8929-5a40d7bbafba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e49f4c77-14f4-4180-8929-5a40d7bbafba-maas-api-openshift-ca\") pod \"e49f4c77-14f4-4180-8929-5a40d7bbafba\" (UID: \"e49f4c77-14f4-4180-8929-5a40d7bbafba\") " Oct 30 20:03:53.614640 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:53.614617 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e49f4c77-14f4-4180-8929-5a40d7bbafba-maas-api-openshift-ca" pod "e49f4c77-14f4-4180-8929-5a40d7bbafba" (UID: "e49f4c77-14f4-4180-8929-5a40d7bbafba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:03:53.614811 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:53.614783 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e49f4c77-14f4-4180-8929-5a40d7bbafba-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:53.616135 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:53.616112 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e49f4c77-14f4-4180-8929-5a40d7bbafba-kube-api-access-x5246" pod "e49f4c77-14f4-4180-8929-5a40d7bbafba" (UID: "e49f4c77-14f4-4180-8929-5a40d7bbafba"). InnerVolumeSpecName "kube-api-access-x5246". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:53.616246 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:53.616231 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e49f4c77-14f4-4180-8929-5a40d7bbafba-maas-api-xks-ca" pod "e49f4c77-14f4-4180-8929-5a40d7bbafba" (UID: "e49f4c77-14f4-4180-8929-5a40d7bbafba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:03:53.716146 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:53.716113 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e49f4c77-14f4-4180-8929-5a40d7bbafba-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:53.716146 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:53.716143 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5246\" (UniqueName: \"kubernetes.io/projected/e49f4c77-14f4-4180-8929-5a40d7bbafba-kube-api-access-x5246\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:53.752908 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:53.752872 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-77a92c-64d9fc577b-pbdb5" podUID="9211eb29-a409-473e-b654-09612e213dc7" containerName="payload-processing" containerID="cri-o://fa5ece7c50fec54ea2d2673bed0fc109a9372d9a38ca83c11fcede402811dde1" gracePeriod=25 Oct 30 20:03:53.870130 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:53.870032 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77a92c-65f6f789f-rxgqj" podUID="5f6986e0-8689-412d-bf77-450b85b0254d" containerName="payload-pre-processing" containerID="cri-o://4c155d2a1b742e13cc2ad4e79fb5795ad6f054fc08a6ea7000ce86bb4277f995" gracePeriod=25 Oct 30 20:03:54.010760 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:54.010732 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-77a92c-64d9fc577b-pbdb5" Oct 30 20:03:54.120619 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:54.120541 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9211eb29-a409-473e-b654-09612e213dc7-kube-api-access-8nh2q\" (UniqueName: \"kubernetes.io/projected/9211eb29-a409-473e-b654-09612e213dc7-kube-api-access-8nh2q\") pod \"9211eb29-a409-473e-b654-09612e213dc7\" (UID: \"9211eb29-a409-473e-b654-09612e213dc7\") " Oct 30 20:03:54.120784 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:54.120627 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9211eb29-a409-473e-b654-09612e213dc7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9211eb29-a409-473e-b654-09612e213dc7-plugins-config-volume\") pod \"9211eb29-a409-473e-b654-09612e213dc7\" (UID: \"9211eb29-a409-473e-b654-09612e213dc7\") " Oct 30 20:03:54.121027 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:54.121001 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9211eb29-a409-473e-b654-09612e213dc7-plugins-config-volume" pod "9211eb29-a409-473e-b654-09612e213dc7" (UID: "9211eb29-a409-473e-b654-09612e213dc7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:03:54.122640 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:54.122620 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9211eb29-a409-473e-b654-09612e213dc7-kube-api-access-8nh2q" pod "9211eb29-a409-473e-b654-09612e213dc7" (UID: "9211eb29-a409-473e-b654-09612e213dc7"). InnerVolumeSpecName "kube-api-access-8nh2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:54.137129 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:54.137092 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77a92c-65f6f789f-rxgqj" Oct 30 20:03:54.222269 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:54.222241 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f6986e0-8689-412d-bf77-450b85b0254d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f6986e0-8689-412d-bf77-450b85b0254d-plugins-config-volume\") pod \"5f6986e0-8689-412d-bf77-450b85b0254d\" (UID: \"5f6986e0-8689-412d-bf77-450b85b0254d\") " Oct 30 20:03:54.222459 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:54.222416 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f6986e0-8689-412d-bf77-450b85b0254d-kube-api-access-zfv8d\" (UniqueName: \"kubernetes.io/projected/5f6986e0-8689-412d-bf77-450b85b0254d-kube-api-access-zfv8d\") pod \"5f6986e0-8689-412d-bf77-450b85b0254d\" (UID: \"5f6986e0-8689-412d-bf77-450b85b0254d\") " Oct 30 20:03:54.222681 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:54.222657 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f6986e0-8689-412d-bf77-450b85b0254d-plugins-config-volume" pod "5f6986e0-8689-412d-bf77-450b85b0254d" (UID: "5f6986e0-8689-412d-bf77-450b85b0254d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:03:54.222758 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:54.222748 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f6986e0-8689-412d-bf77-450b85b0254d-plugins-config-volume\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:54.222800 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:54.222761 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nh2q\" (UniqueName: \"kubernetes.io/projected/9211eb29-a409-473e-b654-09612e213dc7-kube-api-access-8nh2q\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:54.222800 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:54.222770 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9211eb29-a409-473e-b654-09612e213dc7-plugins-config-volume\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:54.224464 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:54.224443 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f6986e0-8689-412d-bf77-450b85b0254d-kube-api-access-zfv8d" pod "5f6986e0-8689-412d-bf77-450b85b0254d" (UID: "5f6986e0-8689-412d-bf77-450b85b0254d"). InnerVolumeSpecName "kube-api-access-zfv8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:54.319989 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:54.319953 4292 generic.go:356] "Generic (PLEG): container finished" podID="9211eb29-a409-473e-b654-09612e213dc7" containerID="fa5ece7c50fec54ea2d2673bed0fc109a9372d9a38ca83c11fcede402811dde1" exitCode=0 Oct 30 20:03:54.320199 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:54.320003 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-77a92c-64d9fc577b-pbdb5" Oct 30 20:03:54.320199 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:54.320044 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-77a92c-64d9fc577b-pbdb5" event={"ID":"9211eb29-a409-473e-b654-09612e213dc7","Type":"ContainerDied","Data":"fa5ece7c50fec54ea2d2673bed0fc109a9372d9a38ca83c11fcede402811dde1"} Oct 30 20:03:54.320199 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:54.320099 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-77a92c-64d9fc577b-pbdb5" event={"ID":"9211eb29-a409-473e-b654-09612e213dc7","Type":"ContainerDied","Data":"ee5241a4b1e0503e2962f74db311d7d141cb7f84bdc35df28c6aa234a7b65c89"} Oct 30 20:03:54.320199 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:54.320124 4292 scope.go:122] "RemoveContainer" containerID="fa5ece7c50fec54ea2d2673bed0fc109a9372d9a38ca83c11fcede402811dde1" Oct 30 20:03:54.322093 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:54.322071 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ffe14b-2vrw7" Oct 30 20:03:54.322194 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:54.322116 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ffe14b-2vrw7" event={"ID":"e49f4c77-14f4-4180-8929-5a40d7bbafba","Type":"ContainerDied","Data":"a5aee78ceb778ac8b610c83eeaa984004406911a80a8cca399541fd6181b4541"} Oct 30 20:03:54.322194 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:54.322176 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5aee78ceb778ac8b610c83eeaa984004406911a80a8cca399541fd6181b4541" Oct 30 20:03:54.323599 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:54.323506 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfv8d\" (UniqueName: \"kubernetes.io/projected/5f6986e0-8689-412d-bf77-450b85b0254d-kube-api-access-zfv8d\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:54.324088 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:54.324063 4292 generic.go:356] "Generic (PLEG): container finished" podID="5f6986e0-8689-412d-bf77-450b85b0254d" containerID="4c155d2a1b742e13cc2ad4e79fb5795ad6f054fc08a6ea7000ce86bb4277f995" exitCode=0 Oct 30 20:03:54.324188 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:54.324108 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77a92c-65f6f789f-rxgqj" event={"ID":"5f6986e0-8689-412d-bf77-450b85b0254d","Type":"ContainerDied","Data":"4c155d2a1b742e13cc2ad4e79fb5795ad6f054fc08a6ea7000ce86bb4277f995"} Oct 30 20:03:54.324188 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:54.324136 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77a92c-65f6f789f-rxgqj" Oct 30 20:03:54.324188 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:54.324159 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77a92c-65f6f789f-rxgqj" event={"ID":"5f6986e0-8689-412d-bf77-450b85b0254d","Type":"ContainerDied","Data":"fd3e6811b36c3d037cc5636dbd0223f5a0972cca95e7f1f2785da0007b2e1371"} Oct 30 20:03:54.331799 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:54.331777 4292 scope.go:122] "RemoveContainer" containerID="fa5ece7c50fec54ea2d2673bed0fc109a9372d9a38ca83c11fcede402811dde1" Oct 30 20:03:54.332119 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:03:54.332097 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa5ece7c50fec54ea2d2673bed0fc109a9372d9a38ca83c11fcede402811dde1\": container with ID starting with fa5ece7c50fec54ea2d2673bed0fc109a9372d9a38ca83c11fcede402811dde1 not found: ID does not exist" containerID="fa5ece7c50fec54ea2d2673bed0fc109a9372d9a38ca83c11fcede402811dde1" Oct 30 20:03:54.332211 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:54.332129 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa5ece7c50fec54ea2d2673bed0fc109a9372d9a38ca83c11fcede402811dde1"} err="failed to get container status \"fa5ece7c50fec54ea2d2673bed0fc109a9372d9a38ca83c11fcede402811dde1\": rpc error: code = NotFound desc = could not find container \"fa5ece7c50fec54ea2d2673bed0fc109a9372d9a38ca83c11fcede402811dde1\": container with ID starting with fa5ece7c50fec54ea2d2673bed0fc109a9372d9a38ca83c11fcede402811dde1 not found: ID does not exist" Oct 30 20:03:54.332211 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:54.332153 4292 scope.go:122] "RemoveContainer" containerID="4c155d2a1b742e13cc2ad4e79fb5795ad6f054fc08a6ea7000ce86bb4277f995" Oct 30 20:03:54.341331 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:54.341309 4292 scope.go:122] "RemoveContainer" containerID="4c155d2a1b742e13cc2ad4e79fb5795ad6f054fc08a6ea7000ce86bb4277f995" Oct 30 20:03:54.341632 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:03:54.341613 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c155d2a1b742e13cc2ad4e79fb5795ad6f054fc08a6ea7000ce86bb4277f995\": container with ID starting with 4c155d2a1b742e13cc2ad4e79fb5795ad6f054fc08a6ea7000ce86bb4277f995 not found: ID does not exist" containerID="4c155d2a1b742e13cc2ad4e79fb5795ad6f054fc08a6ea7000ce86bb4277f995" Oct 30 20:03:54.341696 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:54.341639 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c155d2a1b742e13cc2ad4e79fb5795ad6f054fc08a6ea7000ce86bb4277f995"} err="failed to get container status \"4c155d2a1b742e13cc2ad4e79fb5795ad6f054fc08a6ea7000ce86bb4277f995\": rpc error: code = NotFound desc = could not find container \"4c155d2a1b742e13cc2ad4e79fb5795ad6f054fc08a6ea7000ce86bb4277f995\": container with ID starting with 4c155d2a1b742e13cc2ad4e79fb5795ad6f054fc08a6ea7000ce86bb4277f995 not found: ID does not exist" Oct 30 20:03:54.489683 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:54.489647 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-77a92c-65f6f789f-rxgqj"] Oct 30 20:03:54.492149 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:54.492122 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-77a92c-65f6f789f-rxgqj"] Oct 30 20:03:54.582232 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:54.582192 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f6986e0-8689-412d-bf77-450b85b0254d" path="/var/lib/kubelet/pods/5f6986e0-8689-412d-bf77-450b85b0254d/volumes" Oct 30 20:03:54.657556 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:54.657525 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-77a92c-64d9fc577b-pbdb5"] Oct 30 20:03:54.664259 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:54.664235 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-77a92c-64d9fc577b-pbdb5"] Oct 30 20:03:56.582725 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:56.582696 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9211eb29-a409-473e-b654-09612e213dc7" path="/var/lib/kubelet/pods/9211eb29-a409-473e-b654-09612e213dc7/volumes" Oct 30 20:03:57.152270 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:57.152239 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2"] Oct 30 20:03:57.152483 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:03:57.152459 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2" podUID="ef78c2a5-79c3-45af-aad6-26743703d25e" containerName="istio-proxy" containerID="cri-o://4b4a2c10877b8b2e5492486613299b75a74d389864a2b6fbfda14e4139ff64f7" gracePeriod=30 Oct 30 20:04:00.694025 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:00.693988 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:04:00.694463 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:00.694080 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2" podUID="ef78c2a5-79c3-45af-aad6-26743703d25e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:04:02.376990 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:02.376957 4292 generic.go:356] "Generic (PLEG): container finished" podID="ef78c2a5-79c3-45af-aad6-26743703d25e" containerID="4b4a2c10877b8b2e5492486613299b75a74d389864a2b6fbfda14e4139ff64f7" exitCode=0 Oct 30 20:04:02.377398 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:02.376997 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2" event={"ID":"ef78c2a5-79c3-45af-aad6-26743703d25e","Type":"ContainerDied","Data":"4b4a2c10877b8b2e5492486613299b75a74d389864a2b6fbfda14e4139ff64f7"} Oct 30 20:04:02.412373 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:02.412328 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2" Oct 30 20:04:02.515962 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:02.515880 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef78c2a5-79c3-45af-aad6-26743703d25e-kube-api-access-fxz7f\" (UniqueName: \"kubernetes.io/projected/ef78c2a5-79c3-45af-aad6-26743703d25e-kube-api-access-fxz7f\") pod \"ef78c2a5-79c3-45af-aad6-26743703d25e\" (UID: \"ef78c2a5-79c3-45af-aad6-26743703d25e\") " Oct 30 20:04:02.515962 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:02.515934 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef78c2a5-79c3-45af-aad6-26743703d25e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef78c2a5-79c3-45af-aad6-26743703d25e-workload-socket\") pod \"ef78c2a5-79c3-45af-aad6-26743703d25e\" (UID: \"ef78c2a5-79c3-45af-aad6-26743703d25e\") " Oct 30 20:04:02.516134 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:02.516006 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef78c2a5-79c3-45af-aad6-26743703d25e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef78c2a5-79c3-45af-aad6-26743703d25e-workload-certs\") pod \"ef78c2a5-79c3-45af-aad6-26743703d25e\" (UID: \"ef78c2a5-79c3-45af-aad6-26743703d25e\") " Oct 30 20:04:02.516134 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:02.516064 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef78c2a5-79c3-45af-aad6-26743703d25e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef78c2a5-79c3-45af-aad6-26743703d25e-istio-data\") pod \"ef78c2a5-79c3-45af-aad6-26743703d25e\" (UID: \"ef78c2a5-79c3-45af-aad6-26743703d25e\") " Oct 30 20:04:02.516134 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:02.516089 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ef78c2a5-79c3-45af-aad6-26743703d25e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef78c2a5-79c3-45af-aad6-26743703d25e-istio-podinfo\") pod \"ef78c2a5-79c3-45af-aad6-26743703d25e\" (UID: \"ef78c2a5-79c3-45af-aad6-26743703d25e\") " Oct 30 20:04:02.516272 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:02.516139 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef78c2a5-79c3-45af-aad6-26743703d25e-istio-token\" (UniqueName: \"kubernetes.io/projected/ef78c2a5-79c3-45af-aad6-26743703d25e-istio-token\") pod \"ef78c2a5-79c3-45af-aad6-26743703d25e\" (UID: \"ef78c2a5-79c3-45af-aad6-26743703d25e\") " Oct 30 20:04:02.516272 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:02.516193 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef78c2a5-79c3-45af-aad6-26743703d25e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef78c2a5-79c3-45af-aad6-26743703d25e-istio-envoy\") pod \"ef78c2a5-79c3-45af-aad6-26743703d25e\" (UID: \"ef78c2a5-79c3-45af-aad6-26743703d25e\") " Oct 30 20:04:02.516272 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:02.516242 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef78c2a5-79c3-45af-aad6-26743703d25e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef78c2a5-79c3-45af-aad6-26743703d25e-istiod-ca-cert\") pod \"ef78c2a5-79c3-45af-aad6-26743703d25e\" (UID: \"ef78c2a5-79c3-45af-aad6-26743703d25e\") " Oct 30 20:04:02.516460 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:02.516299 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef78c2a5-79c3-45af-aad6-26743703d25e-workload-socket" pod "ef78c2a5-79c3-45af-aad6-26743703d25e" (UID: "ef78c2a5-79c3-45af-aad6-26743703d25e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:02.516460 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:02.516330 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef78c2a5-79c3-45af-aad6-26743703d25e-istio-data" pod "ef78c2a5-79c3-45af-aad6-26743703d25e" (UID: "ef78c2a5-79c3-45af-aad6-26743703d25e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:02.516460 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:02.516346 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef78c2a5-79c3-45af-aad6-26743703d25e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef78c2a5-79c3-45af-aad6-26743703d25e-credential-socket\") pod \"ef78c2a5-79c3-45af-aad6-26743703d25e\" (UID: \"ef78c2a5-79c3-45af-aad6-26743703d25e\") " Oct 30 20:04:02.516460 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:02.516345 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef78c2a5-79c3-45af-aad6-26743703d25e-workload-certs" pod "ef78c2a5-79c3-45af-aad6-26743703d25e" (UID: "ef78c2a5-79c3-45af-aad6-26743703d25e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:02.516681 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:02.516640 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef78c2a5-79c3-45af-aad6-26743703d25e-credential-socket" pod "ef78c2a5-79c3-45af-aad6-26743703d25e" (UID: "ef78c2a5-79c3-45af-aad6-26743703d25e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:02.516740 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:02.516679 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef78c2a5-79c3-45af-aad6-26743703d25e-istiod-ca-cert" pod "ef78c2a5-79c3-45af-aad6-26743703d25e" (UID: "ef78c2a5-79c3-45af-aad6-26743703d25e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:04:02.516885 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:02.516864 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef78c2a5-79c3-45af-aad6-26743703d25e-workload-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:04:02.516976 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:02.516890 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef78c2a5-79c3-45af-aad6-26743703d25e-istio-data\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:04:02.516976 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:02.516904 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef78c2a5-79c3-45af-aad6-26743703d25e-istiod-ca-cert\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:04:02.516976 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:02.516920 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef78c2a5-79c3-45af-aad6-26743703d25e-credential-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:04:02.516976 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:02.516933 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef78c2a5-79c3-45af-aad6-26743703d25e-workload-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:04:02.518434 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:02.518404 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef78c2a5-79c3-45af-aad6-26743703d25e-istio-envoy" pod "ef78c2a5-79c3-45af-aad6-26743703d25e" (UID: "ef78c2a5-79c3-45af-aad6-26743703d25e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:02.518535 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:02.518435 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef78c2a5-79c3-45af-aad6-26743703d25e-istio-podinfo" pod "ef78c2a5-79c3-45af-aad6-26743703d25e" (UID: "ef78c2a5-79c3-45af-aad6-26743703d25e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:04:02.518660 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:02.518638 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef78c2a5-79c3-45af-aad6-26743703d25e-kube-api-access-fxz7f" pod "ef78c2a5-79c3-45af-aad6-26743703d25e" (UID: "ef78c2a5-79c3-45af-aad6-26743703d25e"). InnerVolumeSpecName "kube-api-access-fxz7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:02.518701 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:02.518649 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef78c2a5-79c3-45af-aad6-26743703d25e-istio-token" pod "ef78c2a5-79c3-45af-aad6-26743703d25e" (UID: "ef78c2a5-79c3-45af-aad6-26743703d25e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:02.618050 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:02.618017 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef78c2a5-79c3-45af-aad6-26743703d25e-istio-podinfo\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:04:02.618050 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:02.618046 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef78c2a5-79c3-45af-aad6-26743703d25e-istio-token\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:04:02.618050 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:02.618057 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef78c2a5-79c3-45af-aad6-26743703d25e-istio-envoy\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:04:02.618326 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:02.618067 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxz7f\" (UniqueName: \"kubernetes.io/projected/ef78c2a5-79c3-45af-aad6-26743703d25e-kube-api-access-fxz7f\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:04:03.383265 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:03.383231 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2" Oct 30 20:04:03.383265 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:03.383244 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2" event={"ID":"ef78c2a5-79c3-45af-aad6-26743703d25e","Type":"ContainerDied","Data":"8bf2e5f38cf38b5a2592045f93a55a58af2280d6feabd58b234fc267e75fdcf2"} Oct 30 20:04:03.383833 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:03.383300 4292 scope.go:122] "RemoveContainer" containerID="4b4a2c10877b8b2e5492486613299b75a74d389864a2b6fbfda14e4139ff64f7" Oct 30 20:04:03.552766 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:03.552734 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2"] Oct 30 20:04:03.603309 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:03.603271 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-bvpp2"] Oct 30 20:04:04.582628 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:04.582599 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef78c2a5-79c3-45af-aad6-26743703d25e" path="/var/lib/kubelet/pods/ef78c2a5-79c3-45af-aad6-26743703d25e/volumes" Oct 30 20:04:11.111073 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:11.110629 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d80c7d-5tvk8"] Oct 30 20:04:11.117485 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:11.111783 4292 cpu_manager.go:414] "containerMap: removing container" podUID="e49f4c77-14f4-4180-8929-5a40d7bbafba" containerName="revoke-keys" Oct 30 20:04:11.117485 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:11.111807 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e49f4c77-14f4-4180-8929-5a40d7bbafba" containerName="revoke-keys" Oct 30 20:04:11.117485 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:11.111839 4292 cpu_manager.go:414] "containerMap: removing container" podUID="5f6986e0-8689-412d-bf77-450b85b0254d" containerName="payload-pre-processing" Oct 30 20:04:11.117485 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:11.111847 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f6986e0-8689-412d-bf77-450b85b0254d" containerName="payload-pre-processing" Oct 30 20:04:11.117485 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:11.111943 4292 cpu_manager.go:414] "containerMap: removing container" podUID="9211eb29-a409-473e-b654-09612e213dc7" containerName="payload-processing" Oct 30 20:04:11.117485 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:11.111950 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9211eb29-a409-473e-b654-09612e213dc7" containerName="payload-processing" Oct 30 20:04:11.117485 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:11.111970 4292 cpu_manager.go:414] "containerMap: removing container" podUID="ef78c2a5-79c3-45af-aad6-26743703d25e" containerName="istio-proxy" Oct 30 20:04:11.117485 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:11.111977 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef78c2a5-79c3-45af-aad6-26743703d25e" containerName="istio-proxy" Oct 30 20:04:11.117485 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:11.112194 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="9211eb29-a409-473e-b654-09612e213dc7" containerName="payload-processing" Oct 30 20:04:11.117485 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:11.112207 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef78c2a5-79c3-45af-aad6-26743703d25e" containerName="istio-proxy" Oct 30 20:04:11.117485 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:11.112223 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="e49f4c77-14f4-4180-8929-5a40d7bbafba" containerName="revoke-keys" Oct 30 20:04:11.117485 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:11.112234 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f6986e0-8689-412d-bf77-450b85b0254d" containerName="payload-pre-processing" Oct 30 20:04:11.117485 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:11.116375 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d80c7d-5tvk8" Oct 30 20:04:11.171014 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:11.170984 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d80c7d-5tvk8"] Oct 30 20:04:11.210823 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:11.210794 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc4f5196-ade3-42ef-8c22-8586adb69246-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d80c7d-5tvk8\" (UID: \"cc4f5196-ade3-42ef-8c22-8586adb69246\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d80c7d-5tvk8" Oct 30 20:04:11.210991 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:11.210838 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc4f5196-ade3-42ef-8c22-8586adb69246-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d80c7d-5tvk8\" (UID: \"cc4f5196-ade3-42ef-8c22-8586adb69246\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d80c7d-5tvk8" Oct 30 20:04:11.210991 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:11.210933 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52nh6\" (UniqueName: \"kubernetes.io/projected/cc4f5196-ade3-42ef-8c22-8586adb69246-kube-api-access-52nh6\") pod \"maas-api-revoke-keys-e2e-ipp-a-d80c7d-5tvk8\" (UID: \"cc4f5196-ade3-42ef-8c22-8586adb69246\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d80c7d-5tvk8" Oct 30 20:04:11.217447 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:11.217425 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:04:11.217560 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:11.217508 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\"" Oct 30 20:04:11.311627 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:11.311596 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc4f5196-ade3-42ef-8c22-8586adb69246-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d80c7d-5tvk8\" (UID: \"cc4f5196-ade3-42ef-8c22-8586adb69246\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d80c7d-5tvk8" Oct 30 20:04:11.311795 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:11.311640 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc4f5196-ade3-42ef-8c22-8586adb69246-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d80c7d-5tvk8\" (UID: \"cc4f5196-ade3-42ef-8c22-8586adb69246\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d80c7d-5tvk8" Oct 30 20:04:11.311795 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:11.311675 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52nh6\" (UniqueName: \"kubernetes.io/projected/cc4f5196-ade3-42ef-8c22-8586adb69246-kube-api-access-52nh6\") pod \"maas-api-revoke-keys-e2e-ipp-a-d80c7d-5tvk8\" (UID: \"cc4f5196-ade3-42ef-8c22-8586adb69246\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d80c7d-5tvk8" Oct 30 20:04:11.312370 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:11.312330 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc4f5196-ade3-42ef-8c22-8586adb69246-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d80c7d-5tvk8\" (UID: \"cc4f5196-ade3-42ef-8c22-8586adb69246\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d80c7d-5tvk8" Oct 30 20:04:11.313955 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:11.313930 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc4f5196-ade3-42ef-8c22-8586adb69246-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d80c7d-5tvk8\" (UID: \"cc4f5196-ade3-42ef-8c22-8586adb69246\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d80c7d-5tvk8" Oct 30 20:04:11.374344 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:11.374274 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52nh6\" (UniqueName: \"kubernetes.io/projected/cc4f5196-ade3-42ef-8c22-8586adb69246-kube-api-access-52nh6\") pod \"maas-api-revoke-keys-e2e-ipp-a-d80c7d-5tvk8\" (UID: \"cc4f5196-ade3-42ef-8c22-8586adb69246\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d80c7d-5tvk8" Oct 30 20:04:11.430457 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:11.430426 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d80c7d-5tvk8" Oct 30 20:04:11.803609 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:11.803574 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d80c7d-5tvk8"] Oct 30 20:04:11.804512 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:04:11.804475 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc4f5196_ade3_42ef_8c22_8586adb69246.slice/crio-030ac6dad3c38cd95a27903bf94988d4f8515bec6d93d88dbbb9feaba5cc9cc7 WatchSource:0}: Error finding container 030ac6dad3c38cd95a27903bf94988d4f8515bec6d93d88dbbb9feaba5cc9cc7: Status 404 returned error can't find the container with id 030ac6dad3c38cd95a27903bf94988d4f8515bec6d93d88dbbb9feaba5cc9cc7 Oct 30 20:04:12.435287 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:12.435194 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d80c7d-5tvk8" event={"ID":"cc4f5196-ade3-42ef-8c22-8586adb69246","Type":"ContainerStarted","Data":"174dbd00c1252887f88f4e7379f7afc2009bb378ae6b59f164dd943b8830d678"} Oct 30 20:04:12.435287 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:12.435253 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d80c7d-5tvk8" event={"ID":"cc4f5196-ade3-42ef-8c22-8586adb69246","Type":"ContainerStarted","Data":"030ac6dad3c38cd95a27903bf94988d4f8515bec6d93d88dbbb9feaba5cc9cc7"} Oct 30 20:04:12.555040 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:12.554992 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d80c7d-5tvk8" podStartSLOduration=1.5549785090000001 podStartE2EDuration="1.554978509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:12.552657979 +0000 UTC m=+2356.070973453" watchObservedRunningTime="2026-10-30 20:04:12.554978509 +0000 UTC m=+2356.073294023" Oct 30 20:04:13.221802 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:04:13.221767 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:04:13.454546 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:13.454507 4292 generic.go:356] "Generic (PLEG): container finished" podID="cc4f5196-ade3-42ef-8c22-8586adb69246" containerID="174dbd00c1252887f88f4e7379f7afc2009bb378ae6b59f164dd943b8830d678" exitCode=0 Oct 30 20:04:13.454962 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:13.454558 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d80c7d-5tvk8" event={"ID":"cc4f5196-ade3-42ef-8c22-8586adb69246","Type":"ContainerDied","Data":"174dbd00c1252887f88f4e7379f7afc2009bb378ae6b59f164dd943b8830d678"} Oct 30 20:04:14.592122 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:14.592097 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d80c7d-5tvk8" Oct 30 20:04:14.653009 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:14.652975 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc4f5196-ade3-42ef-8c22-8586adb69246-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc4f5196-ade3-42ef-8c22-8586adb69246-maas-api-openshift-ca\") pod \"cc4f5196-ade3-42ef-8c22-8586adb69246\" (UID: \"cc4f5196-ade3-42ef-8c22-8586adb69246\") " Oct 30 20:04:14.653195 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:14.653050 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc4f5196-ade3-42ef-8c22-8586adb69246-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc4f5196-ade3-42ef-8c22-8586adb69246-maas-api-xks-ca\") pod \"cc4f5196-ade3-42ef-8c22-8586adb69246\" (UID: \"cc4f5196-ade3-42ef-8c22-8586adb69246\") " Oct 30 20:04:14.653195 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:14.653131 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc4f5196-ade3-42ef-8c22-8586adb69246-kube-api-access-52nh6\" (UniqueName: \"kubernetes.io/projected/cc4f5196-ade3-42ef-8c22-8586adb69246-kube-api-access-52nh6\") pod \"cc4f5196-ade3-42ef-8c22-8586adb69246\" (UID: \"cc4f5196-ade3-42ef-8c22-8586adb69246\") " Oct 30 20:04:14.653415 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:14.653392 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc4f5196-ade3-42ef-8c22-8586adb69246-maas-api-openshift-ca" pod "cc4f5196-ade3-42ef-8c22-8586adb69246" (UID: "cc4f5196-ade3-42ef-8c22-8586adb69246"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:04:14.653663 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:14.653644 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc4f5196-ade3-42ef-8c22-8586adb69246-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:04:14.655210 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:14.655186 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc4f5196-ade3-42ef-8c22-8586adb69246-maas-api-xks-ca" pod "cc4f5196-ade3-42ef-8c22-8586adb69246" (UID: "cc4f5196-ade3-42ef-8c22-8586adb69246"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:04:14.655210 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:14.655193 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc4f5196-ade3-42ef-8c22-8586adb69246-kube-api-access-52nh6" pod "cc4f5196-ade3-42ef-8c22-8586adb69246" (UID: "cc4f5196-ade3-42ef-8c22-8586adb69246"). InnerVolumeSpecName "kube-api-access-52nh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:14.754568 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:14.754492 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc4f5196-ade3-42ef-8c22-8586adb69246-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:04:14.754568 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:14.754518 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52nh6\" (UniqueName: \"kubernetes.io/projected/cc4f5196-ade3-42ef-8c22-8586adb69246-kube-api-access-52nh6\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:04:15.466178 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:15.466146 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d80c7d-5tvk8" Oct 30 20:04:15.466330 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:15.466151 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d80c7d-5tvk8" event={"ID":"cc4f5196-ade3-42ef-8c22-8586adb69246","Type":"ContainerDied","Data":"030ac6dad3c38cd95a27903bf94988d4f8515bec6d93d88dbbb9feaba5cc9cc7"} Oct 30 20:04:15.466330 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:15.466293 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="030ac6dad3c38cd95a27903bf94988d4f8515bec6d93d88dbbb9feaba5cc9cc7" Oct 30 20:04:26.544048 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:26.544014 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a299b8-5hk5j"] Oct 30 20:04:26.544582 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:26.544476 4292 cpu_manager.go:414] "containerMap: removing container" podUID="cc4f5196-ade3-42ef-8c22-8586adb69246" containerName="revoke-keys" Oct 30 20:04:26.544582 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:26.544488 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc4f5196-ade3-42ef-8c22-8586adb69246" containerName="revoke-keys" Oct 30 20:04:26.544661 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:26.544586 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc4f5196-ade3-42ef-8c22-8586adb69246" containerName="revoke-keys" Oct 30 20:04:26.547549 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:26.547530 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a299b8-5hk5j" Oct 30 20:04:26.588036 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:26.588000 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/630aba1d-b9b7-48ff-aa62-690ddd0f61a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a299b8-5hk5j\" (UID: \"630aba1d-b9b7-48ff-aa62-690ddd0f61a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a299b8-5hk5j" Oct 30 20:04:26.588233 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:26.588204 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/630aba1d-b9b7-48ff-aa62-690ddd0f61a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a299b8-5hk5j\" (UID: \"630aba1d-b9b7-48ff-aa62-690ddd0f61a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a299b8-5hk5j" Oct 30 20:04:26.588295 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:26.588258 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drj7h\" (UniqueName: \"kubernetes.io/projected/630aba1d-b9b7-48ff-aa62-690ddd0f61a1-kube-api-access-drj7h\") pod \"maas-api-revoke-keys-e2e-ipp-a-a299b8-5hk5j\" (UID: \"630aba1d-b9b7-48ff-aa62-690ddd0f61a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a299b8-5hk5j" Oct 30 20:04:26.602778 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:26.602749 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a299b8-5hk5j"] Oct 30 20:04:26.650513 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:26.650482 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:04:26.650692 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:26.650619 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\"" Oct 30 20:04:26.689873 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:26.689838 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/630aba1d-b9b7-48ff-aa62-690ddd0f61a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a299b8-5hk5j\" (UID: \"630aba1d-b9b7-48ff-aa62-690ddd0f61a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a299b8-5hk5j" Oct 30 20:04:26.689873 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:26.689876 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drj7h\" (UniqueName: \"kubernetes.io/projected/630aba1d-b9b7-48ff-aa62-690ddd0f61a1-kube-api-access-drj7h\") pod \"maas-api-revoke-keys-e2e-ipp-a-a299b8-5hk5j\" (UID: \"630aba1d-b9b7-48ff-aa62-690ddd0f61a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a299b8-5hk5j" Oct 30 20:04:26.690102 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:26.689994 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/630aba1d-b9b7-48ff-aa62-690ddd0f61a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a299b8-5hk5j\" (UID: \"630aba1d-b9b7-48ff-aa62-690ddd0f61a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a299b8-5hk5j" Oct 30 20:04:26.690677 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:26.690656 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/630aba1d-b9b7-48ff-aa62-690ddd0f61a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a299b8-5hk5j\" (UID: \"630aba1d-b9b7-48ff-aa62-690ddd0f61a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a299b8-5hk5j" Oct 30 20:04:26.692233 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:26.692214 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/630aba1d-b9b7-48ff-aa62-690ddd0f61a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a299b8-5hk5j\" (UID: \"630aba1d-b9b7-48ff-aa62-690ddd0f61a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a299b8-5hk5j" Oct 30 20:04:26.752760 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:26.752725 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drj7h\" (UniqueName: \"kubernetes.io/projected/630aba1d-b9b7-48ff-aa62-690ddd0f61a1-kube-api-access-drj7h\") pod \"maas-api-revoke-keys-e2e-ipp-a-a299b8-5hk5j\" (UID: \"630aba1d-b9b7-48ff-aa62-690ddd0f61a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a299b8-5hk5j" Oct 30 20:04:26.860902 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:26.860826 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a299b8-5hk5j" Oct 30 20:04:27.240396 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:27.240337 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a299b8-5hk5j"] Oct 30 20:04:27.240579 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:04:27.240553 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod630aba1d_b9b7_48ff_aa62_690ddd0f61a1.slice/crio-50af16b936bdf72d465865ca100c0623aaf2e1c7405c5ca1c9aaf444e0fc6f28 WatchSource:0}: Error finding container 50af16b936bdf72d465865ca100c0623aaf2e1c7405c5ca1c9aaf444e0fc6f28: Status 404 returned error can't find the container with id 50af16b936bdf72d465865ca100c0623aaf2e1c7405c5ca1c9aaf444e0fc6f28 Oct 30 20:04:27.529402 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:27.529368 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a299b8-5hk5j" event={"ID":"630aba1d-b9b7-48ff-aa62-690ddd0f61a1","Type":"ContainerStarted","Data":"50af16b936bdf72d465865ca100c0623aaf2e1c7405c5ca1c9aaf444e0fc6f28"} Oct 30 20:04:28.536642 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:28.536601 4292 generic.go:356] "Generic (PLEG): container finished" podID="630aba1d-b9b7-48ff-aa62-690ddd0f61a1" containerID="d43d246151fe2246e8b837de664ae2b46622f3e589de4f6dde4a5ac9c789aeda" exitCode=0 Oct 30 20:04:28.537020 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:28.536647 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a299b8-5hk5j" event={"ID":"630aba1d-b9b7-48ff-aa62-690ddd0f61a1","Type":"ContainerDied","Data":"d43d246151fe2246e8b837de664ae2b46622f3e589de4f6dde4a5ac9c789aeda"} Oct 30 20:04:29.699938 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:29.699905 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a299b8-5hk5j" Oct 30 20:04:29.825476 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:29.825395 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/630aba1d-b9b7-48ff-aa62-690ddd0f61a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/630aba1d-b9b7-48ff-aa62-690ddd0f61a1-maas-api-xks-ca\") pod \"630aba1d-b9b7-48ff-aa62-690ddd0f61a1\" (UID: \"630aba1d-b9b7-48ff-aa62-690ddd0f61a1\") " Oct 30 20:04:29.825644 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:29.825536 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/630aba1d-b9b7-48ff-aa62-690ddd0f61a1-kube-api-access-drj7h\" (UniqueName: \"kubernetes.io/projected/630aba1d-b9b7-48ff-aa62-690ddd0f61a1-kube-api-access-drj7h\") pod \"630aba1d-b9b7-48ff-aa62-690ddd0f61a1\" (UID: \"630aba1d-b9b7-48ff-aa62-690ddd0f61a1\") " Oct 30 20:04:29.825644 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:29.825621 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/630aba1d-b9b7-48ff-aa62-690ddd0f61a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/630aba1d-b9b7-48ff-aa62-690ddd0f61a1-maas-api-openshift-ca\") pod \"630aba1d-b9b7-48ff-aa62-690ddd0f61a1\" (UID: \"630aba1d-b9b7-48ff-aa62-690ddd0f61a1\") " Oct 30 20:04:29.826012 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:29.825988 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/630aba1d-b9b7-48ff-aa62-690ddd0f61a1-maas-api-openshift-ca" pod "630aba1d-b9b7-48ff-aa62-690ddd0f61a1" (UID: "630aba1d-b9b7-48ff-aa62-690ddd0f61a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:04:29.827961 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:29.827937 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/630aba1d-b9b7-48ff-aa62-690ddd0f61a1-maas-api-xks-ca" pod "630aba1d-b9b7-48ff-aa62-690ddd0f61a1" (UID: "630aba1d-b9b7-48ff-aa62-690ddd0f61a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:04:29.828076 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:29.827978 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/630aba1d-b9b7-48ff-aa62-690ddd0f61a1-kube-api-access-drj7h" pod "630aba1d-b9b7-48ff-aa62-690ddd0f61a1" (UID: "630aba1d-b9b7-48ff-aa62-690ddd0f61a1"). InnerVolumeSpecName "kube-api-access-drj7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:29.927416 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:29.927383 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drj7h\" (UniqueName: \"kubernetes.io/projected/630aba1d-b9b7-48ff-aa62-690ddd0f61a1-kube-api-access-drj7h\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:04:29.927416 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:29.927417 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/630aba1d-b9b7-48ff-aa62-690ddd0f61a1-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:04:29.927622 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:29.927435 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/630aba1d-b9b7-48ff-aa62-690ddd0f61a1-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:04:30.548787 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:30.548746 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a299b8-5hk5j" Oct 30 20:04:30.548787 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:30.548772 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a299b8-5hk5j" event={"ID":"630aba1d-b9b7-48ff-aa62-690ddd0f61a1","Type":"ContainerDied","Data":"50af16b936bdf72d465865ca100c0623aaf2e1c7405c5ca1c9aaf444e0fc6f28"} Oct 30 20:04:30.548994 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:30.548824 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50af16b936bdf72d465865ca100c0623aaf2e1c7405c5ca1c9aaf444e0fc6f28" Oct 30 20:04:36.029263 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:36.029227 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c60685-6977c9cbcb-72dxx"] Oct 30 20:04:36.029790 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:36.029774 4292 cpu_manager.go:414] "containerMap: removing container" podUID="630aba1d-b9b7-48ff-aa62-690ddd0f61a1" containerName="revoke-keys" Oct 30 20:04:36.029790 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:36.029791 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="630aba1d-b9b7-48ff-aa62-690ddd0f61a1" containerName="revoke-keys" Oct 30 20:04:36.029878 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:36.029865 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="630aba1d-b9b7-48ff-aa62-690ddd0f61a1" containerName="revoke-keys" Oct 30 20:04:36.033221 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:36.033201 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c60685-6977c9cbcb-72dxx" Oct 30 20:04:36.091542 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:36.091511 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c60685-6977c9cbcb-72dxx"] Oct 30 20:04:36.095412 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:36.095383 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/458e28f7-8eb9-45b9-a6db-aa40888a3e2e-metrics-certs\") pod \"maas-api-e2e-ipp-b-c60685-6977c9cbcb-72dxx\" (UID: \"458e28f7-8eb9-45b9-a6db-aa40888a3e2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c60685-6977c9cbcb-72dxx" Oct 30 20:04:36.095562 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:36.095499 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c5fm\" (UniqueName: \"kubernetes.io/projected/458e28f7-8eb9-45b9-a6db-aa40888a3e2e-kube-api-access-5c5fm\") pod \"maas-api-e2e-ipp-b-c60685-6977c9cbcb-72dxx\" (UID: \"458e28f7-8eb9-45b9-a6db-aa40888a3e2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c60685-6977c9cbcb-72dxx" Oct 30 20:04:36.095643 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:36.095583 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/458e28f7-8eb9-45b9-a6db-aa40888a3e2e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c60685-6977c9cbcb-72dxx\" (UID: \"458e28f7-8eb9-45b9-a6db-aa40888a3e2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c60685-6977c9cbcb-72dxx" Oct 30 20:04:36.095643 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:36.095634 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/458e28f7-8eb9-45b9-a6db-aa40888a3e2e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c60685-6977c9cbcb-72dxx\" (UID: \"458e28f7-8eb9-45b9-a6db-aa40888a3e2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c60685-6977c9cbcb-72dxx" Oct 30 20:04:36.139385 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:36.139344 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-c60685-serving-cert\"" Oct 30 20:04:36.197364 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:36.197306 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/458e28f7-8eb9-45b9-a6db-aa40888a3e2e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c60685-6977c9cbcb-72dxx\" (UID: \"458e28f7-8eb9-45b9-a6db-aa40888a3e2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c60685-6977c9cbcb-72dxx" Oct 30 20:04:36.197529 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:36.197409 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/458e28f7-8eb9-45b9-a6db-aa40888a3e2e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c60685-6977c9cbcb-72dxx\" (UID: \"458e28f7-8eb9-45b9-a6db-aa40888a3e2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c60685-6977c9cbcb-72dxx" Oct 30 20:04:36.197529 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:36.197479 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/458e28f7-8eb9-45b9-a6db-aa40888a3e2e-metrics-certs\") pod \"maas-api-e2e-ipp-b-c60685-6977c9cbcb-72dxx\" (UID: \"458e28f7-8eb9-45b9-a6db-aa40888a3e2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c60685-6977c9cbcb-72dxx" Oct 30 20:04:36.197647 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:36.197551 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c5fm\" (UniqueName: \"kubernetes.io/projected/458e28f7-8eb9-45b9-a6db-aa40888a3e2e-kube-api-access-5c5fm\") pod \"maas-api-e2e-ipp-b-c60685-6977c9cbcb-72dxx\" (UID: \"458e28f7-8eb9-45b9-a6db-aa40888a3e2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c60685-6977c9cbcb-72dxx" Oct 30 20:04:36.197647 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:04:36.197586 4292 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-c60685-serving-cert: secret "maas-api-e2e-ipp-b-c60685-serving-cert" not found Oct 30 20:04:36.197755 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:04:36.197678 4292 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/458e28f7-8eb9-45b9-a6db-aa40888a3e2e-maas-api-tls podName:458e28f7-8eb9-45b9-a6db-aa40888a3e2e nodeName:}" failed. No retries permitted until 2026-10-30 20:04:36.697654155 +0000 UTC m=+2380.215969607 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/458e28f7-8eb9-45b9-a6db-aa40888a3e2e-maas-api-tls") pod "maas-api-e2e-ipp-b-c60685-6977c9cbcb-72dxx" (UID: "458e28f7-8eb9-45b9-a6db-aa40888a3e2e") : secret "maas-api-e2e-ipp-b-c60685-serving-cert" not found Oct 30 20:04:36.198200 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:36.198166 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/458e28f7-8eb9-45b9-a6db-aa40888a3e2e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c60685-6977c9cbcb-72dxx\" (UID: \"458e28f7-8eb9-45b9-a6db-aa40888a3e2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c60685-6977c9cbcb-72dxx" Oct 30 20:04:36.200067 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:36.200044 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/458e28f7-8eb9-45b9-a6db-aa40888a3e2e-metrics-certs\") pod \"maas-api-e2e-ipp-b-c60685-6977c9cbcb-72dxx\" (UID: \"458e28f7-8eb9-45b9-a6db-aa40888a3e2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c60685-6977c9cbcb-72dxx" Oct 30 20:04:36.260741 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:36.260713 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c5fm\" (UniqueName: \"kubernetes.io/projected/458e28f7-8eb9-45b9-a6db-aa40888a3e2e-kube-api-access-5c5fm\") pod \"maas-api-e2e-ipp-b-c60685-6977c9cbcb-72dxx\" (UID: \"458e28f7-8eb9-45b9-a6db-aa40888a3e2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c60685-6977c9cbcb-72dxx" Oct 30 20:04:36.703620 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:36.703579 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/458e28f7-8eb9-45b9-a6db-aa40888a3e2e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c60685-6977c9cbcb-72dxx\" (UID: \"458e28f7-8eb9-45b9-a6db-aa40888a3e2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c60685-6977c9cbcb-72dxx" Oct 30 20:04:36.706170 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:36.706141 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/458e28f7-8eb9-45b9-a6db-aa40888a3e2e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c60685-6977c9cbcb-72dxx\" (UID: \"458e28f7-8eb9-45b9-a6db-aa40888a3e2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c60685-6977c9cbcb-72dxx" Oct 30 20:04:36.946043 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:36.946006 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c60685-6977c9cbcb-72dxx" Oct 30 20:04:37.329205 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:37.329164 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c60685-6977c9cbcb-72dxx"] Oct 30 20:04:37.329705 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:04:37.329680 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod458e28f7_8eb9_45b9_a6db_aa40888a3e2e.slice/crio-f0cb873127ea8aed10e951a1f0c5b4f4f0433d929592e3fe5941150e307b2e0d WatchSource:0}: Error finding container f0cb873127ea8aed10e951a1f0c5b4f4f0433d929592e3fe5941150e307b2e0d: Status 404 returned error can't find the container with id f0cb873127ea8aed10e951a1f0c5b4f4f0433d929592e3fe5941150e307b2e0d Oct 30 20:04:37.331055 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:37.331038 4292 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 20:04:37.591063 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:37.590974 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c60685-6977c9cbcb-72dxx" event={"ID":"458e28f7-8eb9-45b9-a6db-aa40888a3e2e","Type":"ContainerStarted","Data":"f0cb873127ea8aed10e951a1f0c5b4f4f0433d929592e3fe5941150e307b2e0d"} Oct 30 20:04:38.597624 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:38.597538 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c60685-6977c9cbcb-72dxx" event={"ID":"458e28f7-8eb9-45b9-a6db-aa40888a3e2e","Type":"ContainerStarted","Data":"65a6132f89b4d2544d2a2684527407a9bf579e6e556dc5f82a66d250928083ab"} Oct 30 20:04:38.597994 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:38.597722 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c60685-6977c9cbcb-72dxx" Oct 30 20:04:38.718658 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:38.718599 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c60685-6977c9cbcb-72dxx" podStartSLOduration=2.732788479 podStartE2EDuration="3.718583593s" totalImagesPullingTime="985.795114ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:38.717879283 +0000 UTC m=+2382.236194751" watchObservedRunningTime="2026-10-30 20:04:38.718583593 +0000 UTC m=+2382.236899050" Oct 30 20:04:43.220944 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:04:43.220859 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:04:44.611033 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:44.611004 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c60685-6977c9cbcb-72dxx" Oct 30 20:04:47.140406 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:47.140366 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2d2ca9-76879b9f9-97sgs"] Oct 30 20:04:47.144324 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:47.144305 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d2ca9-76879b9f9-97sgs" Oct 30 20:04:47.216651 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:47.216620 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2d2ca9-76879b9f9-97sgs"] Oct 30 20:04:47.244565 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:47.244529 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28jqj\" (UniqueName: \"kubernetes.io/projected/cf9b7462-be83-4c4d-ba6f-5b8f86ab431b-kube-api-access-28jqj\") pod \"payload-pre-processing-e2e-ipp-a-2d2ca9-76879b9f9-97sgs\" (UID: \"cf9b7462-be83-4c4d-ba6f-5b8f86ab431b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d2ca9-76879b9f9-97sgs" Oct 30 20:04:47.244565 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:47.244563 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf9b7462-be83-4c4d-ba6f-5b8f86ab431b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2d2ca9-76879b9f9-97sgs\" (UID: \"cf9b7462-be83-4c4d-ba6f-5b8f86ab431b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d2ca9-76879b9f9-97sgs" Oct 30 20:04:47.264622 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:47.264588 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-2d2ca9\"" Oct 30 20:04:47.264622 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:47.264589 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-2d2ca9-dockercfg-4jl69\"" Oct 30 20:04:47.346194 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:47.346162 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28jqj\" (UniqueName: \"kubernetes.io/projected/cf9b7462-be83-4c4d-ba6f-5b8f86ab431b-kube-api-access-28jqj\") pod \"payload-pre-processing-e2e-ipp-a-2d2ca9-76879b9f9-97sgs\" (UID: \"cf9b7462-be83-4c4d-ba6f-5b8f86ab431b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d2ca9-76879b9f9-97sgs" Oct 30 20:04:47.346393 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:47.346201 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf9b7462-be83-4c4d-ba6f-5b8f86ab431b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2d2ca9-76879b9f9-97sgs\" (UID: \"cf9b7462-be83-4c4d-ba6f-5b8f86ab431b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d2ca9-76879b9f9-97sgs" Oct 30 20:04:47.346928 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:47.346908 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf9b7462-be83-4c4d-ba6f-5b8f86ab431b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2d2ca9-76879b9f9-97sgs\" (UID: \"cf9b7462-be83-4c4d-ba6f-5b8f86ab431b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d2ca9-76879b9f9-97sgs" Oct 30 20:04:47.409292 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:47.409222 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28jqj\" (UniqueName: \"kubernetes.io/projected/cf9b7462-be83-4c4d-ba6f-5b8f86ab431b-kube-api-access-28jqj\") pod \"payload-pre-processing-e2e-ipp-a-2d2ca9-76879b9f9-97sgs\" (UID: \"cf9b7462-be83-4c4d-ba6f-5b8f86ab431b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d2ca9-76879b9f9-97sgs" Oct 30 20:04:47.457055 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:47.457019 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d2ca9-76879b9f9-97sgs" Oct 30 20:04:47.837135 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:47.837096 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2d2ca9-76879b9f9-97sgs"] Oct 30 20:04:47.837944 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:04:47.837917 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf9b7462_be83_4c4d_ba6f_5b8f86ab431b.slice/crio-adf59c507dc50b92230214827d37d56b5d89eaeb212b8011a363dfb354fbe3d0 WatchSource:0}: Error finding container adf59c507dc50b92230214827d37d56b5d89eaeb212b8011a363dfb354fbe3d0: Status 404 returned error can't find the container with id adf59c507dc50b92230214827d37d56b5d89eaeb212b8011a363dfb354fbe3d0 Oct 30 20:04:48.656496 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:48.656459 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d2ca9-76879b9f9-97sgs" event={"ID":"cf9b7462-be83-4c4d-ba6f-5b8f86ab431b","Type":"ContainerStarted","Data":"adf59c507dc50b92230214827d37d56b5d89eaeb212b8011a363dfb354fbe3d0"} Oct 30 20:04:50.669337 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:50.669296 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d2ca9-76879b9f9-97sgs" event={"ID":"cf9b7462-be83-4c4d-ba6f-5b8f86ab431b","Type":"ContainerStarted","Data":"18458718805e25d4812b94bbbb0dec04fdf1ca3e433e5568062998546a8f1075"} Oct 30 20:04:50.669781 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:50.669503 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d2ca9-76879b9f9-97sgs" Oct 30 20:04:50.789860 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:50.789806 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d2ca9-76879b9f9-97sgs" podStartSLOduration=1.076437255 podStartE2EDuration="3.789791973s" totalImagesPullingTime="2.713354718s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:50.78798287 +0000 UTC m=+2394.306298346" watchObservedRunningTime="2026-10-30 20:04:50.789791973 +0000 UTC m=+2394.308107430" Oct 30 20:04:55.556618 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:55.556581 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d80c7d-5tvk8"] Oct 30 20:04:55.605686 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:55.605655 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d80c7d-5tvk8"] Oct 30 20:04:56.588942 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:04:56.588908 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc4f5196-ade3-42ef-8c22-8586adb69246" path="/var/lib/kubelet/pods/cc4f5196-ade3-42ef-8c22-8586adb69246/volumes" Oct 30 20:05:01.687107 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:01.686750 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d2ca9-76879b9f9-97sgs" Oct 30 20:05:02.528374 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:02.524430 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ffe14b-2vrw7"] Oct 30 20:05:02.544045 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:02.543999 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ffe14b-2vrw7"] Oct 30 20:05:02.606704 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:02.606650 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e49f4c77-14f4-4180-8929-5a40d7bbafba" path="/var/lib/kubelet/pods/e49f4c77-14f4-4180-8929-5a40d7bbafba/volumes" Oct 30 20:05:04.005497 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:04.005463 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a299b8-5hk5j"] Oct 30 20:05:04.070376 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:04.061043 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a299b8-5hk5j"] Oct 30 20:05:04.582566 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:04.582525 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="630aba1d-b9b7-48ff-aa62-690ddd0f61a1" path="/var/lib/kubelet/pods/630aba1d-b9b7-48ff-aa62-690ddd0f61a1/volumes" Oct 30 20:05:05.243057 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:05.243021 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8b0edc-8jxln"] Oct 30 20:05:05.248420 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:05.248398 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8b0edc-8jxln" Oct 30 20:05:05.270584 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:05.270553 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w544g\" (UniqueName: \"kubernetes.io/projected/6bcfa73f-0107-460d-bb66-050626157207-kube-api-access-w544g\") pod \"maas-api-revoke-keys-e2e-ipp-b-8b0edc-8jxln\" (UID: \"6bcfa73f-0107-460d-bb66-050626157207\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8b0edc-8jxln" Oct 30 20:05:05.270706 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:05.270593 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bcfa73f-0107-460d-bb66-050626157207-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8b0edc-8jxln\" (UID: \"6bcfa73f-0107-460d-bb66-050626157207\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8b0edc-8jxln" Oct 30 20:05:05.270706 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:05.270608 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bcfa73f-0107-460d-bb66-050626157207-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8b0edc-8jxln\" (UID: \"6bcfa73f-0107-460d-bb66-050626157207\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8b0edc-8jxln" Oct 30 20:05:05.301410 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:05.301382 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8b0edc-8jxln"] Oct 30 20:05:05.349238 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:05.349184 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:05:05.349426 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:05.349271 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\"" Oct 30 20:05:05.371855 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:05.371817 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w544g\" (UniqueName: \"kubernetes.io/projected/6bcfa73f-0107-460d-bb66-050626157207-kube-api-access-w544g\") pod \"maas-api-revoke-keys-e2e-ipp-b-8b0edc-8jxln\" (UID: \"6bcfa73f-0107-460d-bb66-050626157207\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8b0edc-8jxln" Oct 30 20:05:05.371992 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:05.371863 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bcfa73f-0107-460d-bb66-050626157207-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8b0edc-8jxln\" (UID: \"6bcfa73f-0107-460d-bb66-050626157207\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8b0edc-8jxln" Oct 30 20:05:05.371992 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:05.371886 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bcfa73f-0107-460d-bb66-050626157207-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8b0edc-8jxln\" (UID: \"6bcfa73f-0107-460d-bb66-050626157207\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8b0edc-8jxln" Oct 30 20:05:05.372520 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:05.372500 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bcfa73f-0107-460d-bb66-050626157207-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8b0edc-8jxln\" (UID: \"6bcfa73f-0107-460d-bb66-050626157207\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8b0edc-8jxln" Oct 30 20:05:05.374229 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:05.374206 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bcfa73f-0107-460d-bb66-050626157207-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8b0edc-8jxln\" (UID: \"6bcfa73f-0107-460d-bb66-050626157207\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8b0edc-8jxln" Oct 30 20:05:05.434222 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:05.434191 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w544g\" (UniqueName: \"kubernetes.io/projected/6bcfa73f-0107-460d-bb66-050626157207-kube-api-access-w544g\") pod \"maas-api-revoke-keys-e2e-ipp-b-8b0edc-8jxln\" (UID: \"6bcfa73f-0107-460d-bb66-050626157207\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8b0edc-8jxln" Oct 30 20:05:05.566018 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:05.565986 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8b0edc-8jxln" Oct 30 20:05:05.847963 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:05.847934 4292 patch_prober.go:29] interesting pod/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": read tcp 10.133.0.2:43672->10.133.0.98:15021: read: connection reset by peer" start-of-body= Oct 30 20:05:05.848132 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:05.847993 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h" podUID="607ab3e3-4a07-41ef-b1d9-983ddc238675" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": read tcp 10.133.0.2:43672->10.133.0.98:15021: read: connection reset by peer" Oct 30 20:05:05.948274 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:05.948245 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8b0edc-8jxln"] Oct 30 20:05:05.949226 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:05:05.949204 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bcfa73f_0107_460d_bb66_050626157207.slice/crio-9cd902d17dd631b27f471a4a43f70b86f8b18081775eca5a29b3a2e1b5bf3f04 WatchSource:0}: Error finding container 9cd902d17dd631b27f471a4a43f70b86f8b18081775eca5a29b3a2e1b5bf3f04: Status 404 returned error can't find the container with id 9cd902d17dd631b27f471a4a43f70b86f8b18081775eca5a29b3a2e1b5bf3f04 Oct 30 20:05:06.782764 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:06.782724 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8b0edc-8jxln" event={"ID":"6bcfa73f-0107-460d-bb66-050626157207","Type":"ContainerStarted","Data":"fe3a091c7b1afd4b18407013504023d6bafa4269f6548026604a0c6122197686"} Oct 30 20:05:06.783217 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:06.782782 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8b0edc-8jxln" event={"ID":"6bcfa73f-0107-460d-bb66-050626157207","Type":"ContainerStarted","Data":"9cd902d17dd631b27f471a4a43f70b86f8b18081775eca5a29b3a2e1b5bf3f04"} Oct 30 20:05:06.784751 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:06.784727 4292 generic.go:356] "Generic (PLEG): container finished" podID="607ab3e3-4a07-41ef-b1d9-983ddc238675" containerID="de5d656416bcad3602c679675ec8e66f99ba27958850ecdb72f7fe4c58a312bd" exitCode=0 Oct 30 20:05:06.784885 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:06.784806 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h" event={"ID":"607ab3e3-4a07-41ef-b1d9-983ddc238675","Type":"ContainerDied","Data":"de5d656416bcad3602c679675ec8e66f99ba27958850ecdb72f7fe4c58a312bd"} Oct 30 20:05:06.785197 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:06.785180 4292 scope.go:122] "RemoveContainer" containerID="de5d656416bcad3602c679675ec8e66f99ba27958850ecdb72f7fe4c58a312bd" Oct 30 20:05:06.788380 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:06.787307 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:05:06.788380 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:06.787506 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:05:06.788380 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:06.787651 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:05:06.902630 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:06.902579 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8b0edc-8jxln" podStartSLOduration=1.902563515 podStartE2EDuration="1.902563515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:05:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:05:06.901103358 +0000 UTC m=+2410.419418817" watchObservedRunningTime="2026-10-30 20:05:06.902563515 +0000 UTC m=+2410.420878971" Oct 30 20:05:07.792636 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:07.792603 4292 generic.go:356] "Generic (PLEG): container finished" podID="6bcfa73f-0107-460d-bb66-050626157207" containerID="fe3a091c7b1afd4b18407013504023d6bafa4269f6548026604a0c6122197686" exitCode=0 Oct 30 20:05:07.793125 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:07.792691 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8b0edc-8jxln" event={"ID":"6bcfa73f-0107-460d-bb66-050626157207","Type":"ContainerDied","Data":"fe3a091c7b1afd4b18407013504023d6bafa4269f6548026604a0c6122197686"} Oct 30 20:05:07.794798 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:07.794775 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h" event={"ID":"607ab3e3-4a07-41ef-b1d9-983ddc238675","Type":"ContainerStarted","Data":"b397e12c0686dbb626c951e53d9345ec67cfb80586e67be88595f77d3d4b4358"} Oct 30 20:05:08.632445 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:08.632413 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h" Oct 30 20:05:08.632857 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:08.632836 4292 patch_prober.go:29] interesting pod/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 20:05:08.632918 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:08.632892 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h" podUID="607ab3e3-4a07-41ef-b1d9-983ddc238675" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 20:05:09.116724 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:09.116382 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8b0edc-8jxln" Oct 30 20:05:09.156797 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:09.156754 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bcfa73f-0107-460d-bb66-050626157207-kube-api-access-w544g\" (UniqueName: \"kubernetes.io/projected/6bcfa73f-0107-460d-bb66-050626157207-kube-api-access-w544g\") pod \"6bcfa73f-0107-460d-bb66-050626157207\" (UID: \"6bcfa73f-0107-460d-bb66-050626157207\") " Oct 30 20:05:09.157557 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:09.157494 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bcfa73f-0107-460d-bb66-050626157207-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bcfa73f-0107-460d-bb66-050626157207-maas-api-openshift-ca\") pod \"6bcfa73f-0107-460d-bb66-050626157207\" (UID: \"6bcfa73f-0107-460d-bb66-050626157207\") " Oct 30 20:05:09.158485 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:09.158462 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bcfa73f-0107-460d-bb66-050626157207-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bcfa73f-0107-460d-bb66-050626157207-maas-api-xks-ca\") pod \"6bcfa73f-0107-460d-bb66-050626157207\" (UID: \"6bcfa73f-0107-460d-bb66-050626157207\") " Oct 30 20:05:09.159377 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:09.158202 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bcfa73f-0107-460d-bb66-050626157207-maas-api-openshift-ca" pod "6bcfa73f-0107-460d-bb66-050626157207" (UID: "6bcfa73f-0107-460d-bb66-050626157207"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:05:09.200577 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:09.193896 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bcfa73f-0107-460d-bb66-050626157207-maas-api-xks-ca" pod "6bcfa73f-0107-460d-bb66-050626157207" (UID: "6bcfa73f-0107-460d-bb66-050626157207"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:05:09.200577 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:09.197583 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bcfa73f-0107-460d-bb66-050626157207-kube-api-access-w544g" pod "6bcfa73f-0107-460d-bb66-050626157207" (UID: "6bcfa73f-0107-460d-bb66-050626157207"). InnerVolumeSpecName "kube-api-access-w544g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:05:09.262283 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:09.262186 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bcfa73f-0107-460d-bb66-050626157207-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:05:09.262283 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:09.262227 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w544g\" (UniqueName: \"kubernetes.io/projected/6bcfa73f-0107-460d-bb66-050626157207-kube-api-access-w544g\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:05:09.262283 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:09.262243 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bcfa73f-0107-460d-bb66-050626157207-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:05:09.632565 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:09.632123 4292 patch_prober.go:29] interesting pod/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 20:05:09.632565 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:09.632178 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h" podUID="607ab3e3-4a07-41ef-b1d9-983ddc238675" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 20:05:09.834298 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:09.834267 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8b0edc-8jxln" event={"ID":"6bcfa73f-0107-460d-bb66-050626157207","Type":"ContainerDied","Data":"9cd902d17dd631b27f471a4a43f70b86f8b18081775eca5a29b3a2e1b5bf3f04"} Oct 30 20:05:09.834487 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:09.834317 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9cd902d17dd631b27f471a4a43f70b86f8b18081775eca5a29b3a2e1b5bf3f04" Oct 30 20:05:09.834487 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:09.834384 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8b0edc-8jxln" Oct 30 20:05:10.631776 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:10.631740 4292 patch_prober.go:29] interesting pod/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 20:05:10.632183 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:10.631801 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h" podUID="607ab3e3-4a07-41ef-b1d9-983ddc238675" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 20:05:10.733229 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:10.733184 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6"] Oct 30 20:05:10.733837 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:10.733792 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6" podUID="554fd8be-cebc-4035-840e-e377169383e5" containerName="istio-proxy" containerID="cri-o://05237ba0c0e7a3109e8cb1ccde90bfc76424465f898d28d34e12224021a3f57b" gracePeriod=30 Oct 30 20:05:11.617669 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:11.617633 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-77a92c-m6jml"] Oct 30 20:05:11.618120 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:11.618107 4292 cpu_manager.go:414] "containerMap: removing container" podUID="6bcfa73f-0107-460d-bb66-050626157207" containerName="revoke-keys" Oct 30 20:05:11.618174 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:11.618122 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bcfa73f-0107-460d-bb66-050626157207" containerName="revoke-keys" Oct 30 20:05:11.618213 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:11.618207 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bcfa73f-0107-460d-bb66-050626157207" containerName="revoke-keys" Oct 30 20:05:11.620395 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:11.620378 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-77a92c-m6jml" Oct 30 20:05:11.631978 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:11.631955 4292 patch_prober.go:29] interesting pod/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 20:05:11.632274 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:11.631995 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h" podUID="607ab3e3-4a07-41ef-b1d9-983ddc238675" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 20:05:11.679700 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:11.679655 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-77a92c-m6jml"] Oct 30 20:05:11.745407 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:11.734092 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:05:11.758174 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:11.753469 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\"" Oct 30 20:05:11.772841 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:11.772635 4292 patch_prober.go:29] interesting pod/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:05:11.773161 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:11.773109 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6" podUID="554fd8be-cebc-4035-840e-e377169383e5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:05:11.834308 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:11.834276 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6211173e-2380-427d-9501-bd4708e62784-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-77a92c-m6jml\" (UID: \"6211173e-2380-427d-9501-bd4708e62784\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-77a92c-m6jml" Oct 30 20:05:11.834308 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:11.834311 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht4l6\" (UniqueName: \"kubernetes.io/projected/6211173e-2380-427d-9501-bd4708e62784-kube-api-access-ht4l6\") pod \"maas-api-revoke-keys-e2e-ipp-b-77a92c-m6jml\" (UID: \"6211173e-2380-427d-9501-bd4708e62784\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-77a92c-m6jml" Oct 30 20:05:11.834548 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:11.834470 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6211173e-2380-427d-9501-bd4708e62784-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-77a92c-m6jml\" (UID: \"6211173e-2380-427d-9501-bd4708e62784\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-77a92c-m6jml" Oct 30 20:05:11.941272 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:11.941174 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6211173e-2380-427d-9501-bd4708e62784-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-77a92c-m6jml\" (UID: \"6211173e-2380-427d-9501-bd4708e62784\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-77a92c-m6jml" Oct 30 20:05:11.962578 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:11.953861 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6211173e-2380-427d-9501-bd4708e62784-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-77a92c-m6jml\" (UID: \"6211173e-2380-427d-9501-bd4708e62784\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-77a92c-m6jml" Oct 30 20:05:11.962916 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:11.962892 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht4l6\" (UniqueName: \"kubernetes.io/projected/6211173e-2380-427d-9501-bd4708e62784-kube-api-access-ht4l6\") pod \"maas-api-revoke-keys-e2e-ipp-b-77a92c-m6jml\" (UID: \"6211173e-2380-427d-9501-bd4708e62784\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-77a92c-m6jml" Oct 30 20:05:11.963302 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:11.963276 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6211173e-2380-427d-9501-bd4708e62784-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-77a92c-m6jml\" (UID: \"6211173e-2380-427d-9501-bd4708e62784\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-77a92c-m6jml" Oct 30 20:05:12.003302 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:12.001473 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6211173e-2380-427d-9501-bd4708e62784-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-77a92c-m6jml\" (UID: \"6211173e-2380-427d-9501-bd4708e62784\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-77a92c-m6jml" Oct 30 20:05:12.026494 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:12.026467 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht4l6\" (UniqueName: \"kubernetes.io/projected/6211173e-2380-427d-9501-bd4708e62784-kube-api-access-ht4l6\") pod \"maas-api-revoke-keys-e2e-ipp-b-77a92c-m6jml\" (UID: \"6211173e-2380-427d-9501-bd4708e62784\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-77a92c-m6jml" Oct 30 20:05:12.232531 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:12.232439 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-77a92c-m6jml" Oct 30 20:05:12.632324 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:12.632294 4292 patch_prober.go:29] interesting pod/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 20:05:12.632743 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:12.632345 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h" podUID="607ab3e3-4a07-41ef-b1d9-983ddc238675" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 20:05:12.633716 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:12.633693 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-77a92c-m6jml"] Oct 30 20:05:12.634311 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:05:12.634292 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6211173e_2380_427d_9501_bd4708e62784.slice/crio-66e550b6191e2200b2bf255186c0c5db43032e4ca2e4a4f4c9d32011869f980a WatchSource:0}: Error finding container 66e550b6191e2200b2bf255186c0c5db43032e4ca2e4a4f4c9d32011869f980a: Status 404 returned error can't find the container with id 66e550b6191e2200b2bf255186c0c5db43032e4ca2e4a4f4c9d32011869f980a Oct 30 20:05:12.928704 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:12.928611 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-77a92c-m6jml" event={"ID":"6211173e-2380-427d-9501-bd4708e62784","Type":"ContainerStarted","Data":"66e550b6191e2200b2bf255186c0c5db43032e4ca2e4a4f4c9d32011869f980a"} Oct 30 20:05:13.222852 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:05:13.222762 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:05:13.636510 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:13.636481 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h" Oct 30 20:05:13.637003 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:13.636883 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h" Oct 30 20:05:13.637529 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:13.637508 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h" Oct 30 20:05:13.937551 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:13.937452 4292 generic.go:356] "Generic (PLEG): container finished" podID="6211173e-2380-427d-9501-bd4708e62784" containerID="dcb4f3ec7860cb84f9a5c4c742c0908e7c63d2a03810b408a4a61ef97da3d59e" exitCode=0 Oct 30 20:05:13.937784 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:13.937639 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-77a92c-m6jml" event={"ID":"6211173e-2380-427d-9501-bd4708e62784","Type":"ContainerDied","Data":"dcb4f3ec7860cb84f9a5c4c742c0908e7c63d2a03810b408a4a61ef97da3d59e"} Oct 30 20:05:15.094562 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:15.094537 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-77a92c-m6jml" Oct 30 20:05:15.209256 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:15.209114 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6211173e-2380-427d-9501-bd4708e62784-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6211173e-2380-427d-9501-bd4708e62784-maas-api-openshift-ca\") pod \"6211173e-2380-427d-9501-bd4708e62784\" (UID: \"6211173e-2380-427d-9501-bd4708e62784\") " Oct 30 20:05:15.209463 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:15.209253 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6211173e-2380-427d-9501-bd4708e62784-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6211173e-2380-427d-9501-bd4708e62784-maas-api-xks-ca\") pod \"6211173e-2380-427d-9501-bd4708e62784\" (UID: \"6211173e-2380-427d-9501-bd4708e62784\") " Oct 30 20:05:15.209463 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:15.209374 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6211173e-2380-427d-9501-bd4708e62784-kube-api-access-ht4l6\" (UniqueName: \"kubernetes.io/projected/6211173e-2380-427d-9501-bd4708e62784-kube-api-access-ht4l6\") pod \"6211173e-2380-427d-9501-bd4708e62784\" (UID: \"6211173e-2380-427d-9501-bd4708e62784\") " Oct 30 20:05:15.209654 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:15.209628 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6211173e-2380-427d-9501-bd4708e62784-maas-api-openshift-ca" pod "6211173e-2380-427d-9501-bd4708e62784" (UID: "6211173e-2380-427d-9501-bd4708e62784"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:05:15.209950 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:15.209883 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6211173e-2380-427d-9501-bd4708e62784-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:05:15.211846 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:15.211813 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6211173e-2380-427d-9501-bd4708e62784-maas-api-xks-ca" pod "6211173e-2380-427d-9501-bd4708e62784" (UID: "6211173e-2380-427d-9501-bd4708e62784"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:05:15.211961 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:15.211941 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6211173e-2380-427d-9501-bd4708e62784-kube-api-access-ht4l6" pod "6211173e-2380-427d-9501-bd4708e62784" (UID: "6211173e-2380-427d-9501-bd4708e62784"). InnerVolumeSpecName "kube-api-access-ht4l6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:05:15.310693 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:15.310662 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6211173e-2380-427d-9501-bd4708e62784-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:05:15.310693 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:15.310689 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht4l6\" (UniqueName: \"kubernetes.io/projected/6211173e-2380-427d-9501-bd4708e62784-kube-api-access-ht4l6\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:05:15.951767 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:15.951724 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-77a92c-m6jml" event={"ID":"6211173e-2380-427d-9501-bd4708e62784","Type":"ContainerDied","Data":"66e550b6191e2200b2bf255186c0c5db43032e4ca2e4a4f4c9d32011869f980a"} Oct 30 20:05:15.951954 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:15.951792 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66e550b6191e2200b2bf255186c0c5db43032e4ca2e4a4f4c9d32011869f980a" Oct 30 20:05:15.951954 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:15.951864 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-77a92c-m6jml" Oct 30 20:05:15.954907 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:15.954875 4292 generic.go:356] "Generic (PLEG): container finished" podID="554fd8be-cebc-4035-840e-e377169383e5" containerID="05237ba0c0e7a3109e8cb1ccde90bfc76424465f898d28d34e12224021a3f57b" exitCode=0 Oct 30 20:05:15.955054 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:15.954955 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6" event={"ID":"554fd8be-cebc-4035-840e-e377169383e5","Type":"ContainerDied","Data":"05237ba0c0e7a3109e8cb1ccde90bfc76424465f898d28d34e12224021a3f57b"} Oct 30 20:05:15.955054 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:15.955008 4292 scope.go:122] "RemoveContainer" containerID="4d37c3a61cf245489e3daf66c689410e3d7155c5ed1263cb202415cdd7d6d768" Oct 30 20:05:15.978749 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:15.978729 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6" Oct 30 20:05:16.120159 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:16.120135 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/554fd8be-cebc-4035-840e-e377169383e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/554fd8be-cebc-4035-840e-e377169383e5-istio-podinfo\") pod \"554fd8be-cebc-4035-840e-e377169383e5\" (UID: \"554fd8be-cebc-4035-840e-e377169383e5\") " Oct 30 20:05:16.120627 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:16.120194 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/554fd8be-cebc-4035-840e-e377169383e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/554fd8be-cebc-4035-840e-e377169383e5-istio-data\") pod \"554fd8be-cebc-4035-840e-e377169383e5\" (UID: \"554fd8be-cebc-4035-840e-e377169383e5\") " Oct 30 20:05:16.120627 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:16.120216 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/554fd8be-cebc-4035-840e-e377169383e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/554fd8be-cebc-4035-840e-e377169383e5-credential-socket\") pod \"554fd8be-cebc-4035-840e-e377169383e5\" (UID: \"554fd8be-cebc-4035-840e-e377169383e5\") " Oct 30 20:05:16.120627 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:16.120315 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/554fd8be-cebc-4035-840e-e377169383e5-istio-token\" (UniqueName: \"kubernetes.io/projected/554fd8be-cebc-4035-840e-e377169383e5-istio-token\") pod \"554fd8be-cebc-4035-840e-e377169383e5\" (UID: \"554fd8be-cebc-4035-840e-e377169383e5\") " Oct 30 20:05:16.120627 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:16.120470 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/554fd8be-cebc-4035-840e-e377169383e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/554fd8be-cebc-4035-840e-e377169383e5-workload-certs\") pod \"554fd8be-cebc-4035-840e-e377169383e5\" (UID: \"554fd8be-cebc-4035-840e-e377169383e5\") " Oct 30 20:05:16.120627 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:16.120480 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/554fd8be-cebc-4035-840e-e377169383e5-credential-socket" pod "554fd8be-cebc-4035-840e-e377169383e5" (UID: "554fd8be-cebc-4035-840e-e377169383e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:05:16.120627 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:16.120519 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/554fd8be-cebc-4035-840e-e377169383e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/554fd8be-cebc-4035-840e-e377169383e5-istiod-ca-cert\") pod \"554fd8be-cebc-4035-840e-e377169383e5\" (UID: \"554fd8be-cebc-4035-840e-e377169383e5\") " Oct 30 20:05:16.120627 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:16.120546 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/554fd8be-cebc-4035-840e-e377169383e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/554fd8be-cebc-4035-840e-e377169383e5-istio-envoy\") pod \"554fd8be-cebc-4035-840e-e377169383e5\" (UID: \"554fd8be-cebc-4035-840e-e377169383e5\") " Oct 30 20:05:16.120627 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:16.120564 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/554fd8be-cebc-4035-840e-e377169383e5-istio-data" pod "554fd8be-cebc-4035-840e-e377169383e5" (UID: "554fd8be-cebc-4035-840e-e377169383e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:05:16.120627 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:16.120594 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/554fd8be-cebc-4035-840e-e377169383e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/554fd8be-cebc-4035-840e-e377169383e5-workload-socket\") pod \"554fd8be-cebc-4035-840e-e377169383e5\" (UID: \"554fd8be-cebc-4035-840e-e377169383e5\") " Oct 30 20:05:16.120627 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:16.120620 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/554fd8be-cebc-4035-840e-e377169383e5-kube-api-access-6kckt\" (UniqueName: \"kubernetes.io/projected/554fd8be-cebc-4035-840e-e377169383e5-kube-api-access-6kckt\") pod \"554fd8be-cebc-4035-840e-e377169383e5\" (UID: \"554fd8be-cebc-4035-840e-e377169383e5\") " Oct 30 20:05:16.121159 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:16.120681 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/554fd8be-cebc-4035-840e-e377169383e5-workload-certs" pod "554fd8be-cebc-4035-840e-e377169383e5" (UID: "554fd8be-cebc-4035-840e-e377169383e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:05:16.121159 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:16.120881 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/554fd8be-cebc-4035-840e-e377169383e5-workload-socket" pod "554fd8be-cebc-4035-840e-e377169383e5" (UID: "554fd8be-cebc-4035-840e-e377169383e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:05:16.121159 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:16.120907 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/554fd8be-cebc-4035-840e-e377169383e5-istiod-ca-cert" pod "554fd8be-cebc-4035-840e-e377169383e5" (UID: "554fd8be-cebc-4035-840e-e377169383e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:05:16.121159 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:16.121103 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/554fd8be-cebc-4035-840e-e377169383e5-workload-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:05:16.121159 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:16.121117 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/554fd8be-cebc-4035-840e-e377169383e5-istiod-ca-cert\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:05:16.121159 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:16.121127 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/554fd8be-cebc-4035-840e-e377169383e5-workload-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:05:16.121159 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:16.121138 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/554fd8be-cebc-4035-840e-e377169383e5-istio-data\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:05:16.121159 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:16.121150 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/554fd8be-cebc-4035-840e-e377169383e5-credential-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:05:16.122722 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:16.122696 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/554fd8be-cebc-4035-840e-e377169383e5-istio-podinfo" pod "554fd8be-cebc-4035-840e-e377169383e5" (UID: "554fd8be-cebc-4035-840e-e377169383e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:05:16.122722 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:16.122708 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/554fd8be-cebc-4035-840e-e377169383e5-kube-api-access-6kckt" pod "554fd8be-cebc-4035-840e-e377169383e5" (UID: "554fd8be-cebc-4035-840e-e377169383e5"). InnerVolumeSpecName "kube-api-access-6kckt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:05:16.122959 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:16.122939 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/554fd8be-cebc-4035-840e-e377169383e5-istio-envoy" pod "554fd8be-cebc-4035-840e-e377169383e5" (UID: "554fd8be-cebc-4035-840e-e377169383e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:05:16.122959 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:16.122953 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/554fd8be-cebc-4035-840e-e377169383e5-istio-token" pod "554fd8be-cebc-4035-840e-e377169383e5" (UID: "554fd8be-cebc-4035-840e-e377169383e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:05:16.221824 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:16.221791 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kckt\" (UniqueName: \"kubernetes.io/projected/554fd8be-cebc-4035-840e-e377169383e5-kube-api-access-6kckt\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:05:16.221824 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:16.221819 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/554fd8be-cebc-4035-840e-e377169383e5-istio-podinfo\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:05:16.221824 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:16.221830 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/554fd8be-cebc-4035-840e-e377169383e5-istio-token\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:05:16.222057 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:16.221841 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/554fd8be-cebc-4035-840e-e377169383e5-istio-envoy\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:05:16.965602 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:16.965519 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6" Oct 30 20:05:16.965602 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:16.965538 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6" event={"ID":"554fd8be-cebc-4035-840e-e377169383e5","Type":"ContainerDied","Data":"c470489af1bedccf2346d2dc5c4b6a2df788ab28016e5f7560b68e2f73823736"} Oct 30 20:05:16.965602 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:16.965594 4292 scope.go:122] "RemoveContainer" containerID="05237ba0c0e7a3109e8cb1ccde90bfc76424465f898d28d34e12224021a3f57b" Oct 30 20:05:17.188524 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:17.188489 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6"] Oct 30 20:05:17.239514 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:17.239427 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-ffe14b-openshift-default-848dcb67b7-j7bs6"] Oct 30 20:05:18.582747 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:05:18.582717 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="554fd8be-cebc-4035-840e-e377169383e5" path="/var/lib/kubelet/pods/554fd8be-cebc-4035-840e-e377169383e5/volumes" Oct 30 20:05:43.299552 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:05:43.299509 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:06:06.310172 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:06.310136 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c60685-5d446879b7-d72xw"] Oct 30 20:06:11.356705 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:11.356600 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c60685-5d446879b7-d72xw" podUID="9ec677ce-8380-4793-b6cf-02a092f9ffa5" containerName="payload-pre-processing" containerID="cri-o://de2ddfb345a683bea6ff906963431b1d7521d76ab825f20176d2de0d6abd6626" gracePeriod=25 Oct 30 20:06:11.619900 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:11.619839 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c60685-5d446879b7-d72xw" Oct 30 20:06:11.707661 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:11.707632 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ec677ce-8380-4793-b6cf-02a092f9ffa5-kube-api-access-kxdcm\" (UniqueName: \"kubernetes.io/projected/9ec677ce-8380-4793-b6cf-02a092f9ffa5-kube-api-access-kxdcm\") pod \"9ec677ce-8380-4793-b6cf-02a092f9ffa5\" (UID: \"9ec677ce-8380-4793-b6cf-02a092f9ffa5\") " Oct 30 20:06:11.707661 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:11.707664 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ec677ce-8380-4793-b6cf-02a092f9ffa5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ec677ce-8380-4793-b6cf-02a092f9ffa5-plugins-config-volume\") pod \"9ec677ce-8380-4793-b6cf-02a092f9ffa5\" (UID: \"9ec677ce-8380-4793-b6cf-02a092f9ffa5\") " Oct 30 20:06:11.708037 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:11.708016 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ec677ce-8380-4793-b6cf-02a092f9ffa5-plugins-config-volume" pod "9ec677ce-8380-4793-b6cf-02a092f9ffa5" (UID: "9ec677ce-8380-4793-b6cf-02a092f9ffa5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:06:11.709815 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:11.709791 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ec677ce-8380-4793-b6cf-02a092f9ffa5-kube-api-access-kxdcm" pod "9ec677ce-8380-4793-b6cf-02a092f9ffa5" (UID: "9ec677ce-8380-4793-b6cf-02a092f9ffa5"). InnerVolumeSpecName "kube-api-access-kxdcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:06:11.809751 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:11.809715 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxdcm\" (UniqueName: \"kubernetes.io/projected/9ec677ce-8380-4793-b6cf-02a092f9ffa5-kube-api-access-kxdcm\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:11.809751 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:11.809745 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ec677ce-8380-4793-b6cf-02a092f9ffa5-plugins-config-volume\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:12.390004 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:12.389969 4292 generic.go:356] "Generic (PLEG): container finished" podID="9ec677ce-8380-4793-b6cf-02a092f9ffa5" containerID="de2ddfb345a683bea6ff906963431b1d7521d76ab825f20176d2de0d6abd6626" exitCode=0 Oct 30 20:06:12.390472 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:12.390018 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c60685-5d446879b7-d72xw" event={"ID":"9ec677ce-8380-4793-b6cf-02a092f9ffa5","Type":"ContainerDied","Data":"de2ddfb345a683bea6ff906963431b1d7521d76ab825f20176d2de0d6abd6626"} Oct 30 20:06:12.390472 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:12.390021 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c60685-5d446879b7-d72xw" Oct 30 20:06:12.390472 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:12.390061 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c60685-5d446879b7-d72xw" event={"ID":"9ec677ce-8380-4793-b6cf-02a092f9ffa5","Type":"ContainerDied","Data":"5d30ae0b6f181be062b59ff43560eff78c50b8a577c0ef226c3e3d79c37900fa"} Oct 30 20:06:12.390472 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:12.390082 4292 scope.go:122] "RemoveContainer" containerID="de2ddfb345a683bea6ff906963431b1d7521d76ab825f20176d2de0d6abd6626" Oct 30 20:06:12.400021 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:12.400002 4292 scope.go:122] "RemoveContainer" containerID="de2ddfb345a683bea6ff906963431b1d7521d76ab825f20176d2de0d6abd6626" Oct 30 20:06:12.400273 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:06:12.400253 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de2ddfb345a683bea6ff906963431b1d7521d76ab825f20176d2de0d6abd6626\": container with ID starting with de2ddfb345a683bea6ff906963431b1d7521d76ab825f20176d2de0d6abd6626 not found: ID does not exist" containerID="de2ddfb345a683bea6ff906963431b1d7521d76ab825f20176d2de0d6abd6626" Oct 30 20:06:12.400319 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:12.400282 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de2ddfb345a683bea6ff906963431b1d7521d76ab825f20176d2de0d6abd6626"} err="failed to get container status \"de2ddfb345a683bea6ff906963431b1d7521d76ab825f20176d2de0d6abd6626\": rpc error: code = NotFound desc = could not find container \"de2ddfb345a683bea6ff906963431b1d7521d76ab825f20176d2de0d6abd6626\": container with ID starting with de2ddfb345a683bea6ff906963431b1d7521d76ab825f20176d2de0d6abd6626 not found: ID does not exist" Oct 30 20:06:12.559421 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:12.559391 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c60685-5d446879b7-d72xw"] Oct 30 20:06:12.610061 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:12.610029 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c60685-5d446879b7-d72xw"] Oct 30 20:06:13.220699 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:06:13.220659 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:06:14.582705 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:14.582676 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ec677ce-8380-4793-b6cf-02a092f9ffa5" path="/var/lib/kubelet/pods/9ec677ce-8380-4793-b6cf-02a092f9ffa5/volumes" Oct 30 20:06:20.454214 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:20.450908 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-77a92c-m6jml"] Oct 30 20:06:20.469802 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:20.469763 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-77a92c-m6jml"] Oct 30 20:06:20.581920 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:20.581886 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6211173e-2380-427d-9501-bd4708e62784" path="/var/lib/kubelet/pods/6211173e-2380-427d-9501-bd4708e62784/volumes" Oct 30 20:06:21.751452 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:21.751417 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8b0edc-8jxln"] Oct 30 20:06:21.802163 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:21.802135 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8b0edc-8jxln"] Oct 30 20:06:22.582739 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:22.582701 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bcfa73f-0107-460d-bb66-050626157207" path="/var/lib/kubelet/pods/6bcfa73f-0107-460d-bb66-050626157207/volumes" Oct 30 20:06:29.215040 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:29.215003 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr"] Oct 30 20:06:29.215645 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:29.215264 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr" podUID="cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe" containerName="istio-proxy" containerID="cri-o://9f19d7f1745cd961b77bab3ee2f4be2bd854c951b726fdc08ff73058a6c196fd" gracePeriod=30 Oct 30 20:06:34.473929 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:34.473904 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr" Oct 30 20:06:34.507795 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:34.507733 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe-workload-socket\") pod \"cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe\" (UID: \"cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe\") " Oct 30 20:06:34.507912 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:34.507827 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe-istio-envoy\") pod \"cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe\" (UID: \"cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe\") " Oct 30 20:06:34.507912 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:34.507851 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe-kube-api-access-plj6d\" (UniqueName: \"kubernetes.io/projected/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe-kube-api-access-plj6d\") pod \"cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe\" (UID: \"cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe\") " Oct 30 20:06:34.507912 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:34.507872 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe-credential-socket\") pod \"cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe\" (UID: \"cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe\") " Oct 30 20:06:34.508054 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:34.507912 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe-istio-data\") pod \"cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe\" (UID: \"cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe\") " Oct 30 20:06:34.508054 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:34.507996 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe-workload-socket" pod "cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe" (UID: "cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:06:34.508054 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:34.508034 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe-istiod-ca-cert\") pod \"cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe\" (UID: \"cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe\") " Oct 30 20:06:34.508192 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:34.508062 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe-istio-podinfo\") pod \"cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe\" (UID: \"cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe\") " Oct 30 20:06:34.508192 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:34.508088 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe-istio-token\" (UniqueName: \"kubernetes.io/projected/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe-istio-token\") pod \"cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe\" (UID: \"cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe\") " Oct 30 20:06:34.508192 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:34.508128 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe-workload-certs\") pod \"cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe\" (UID: \"cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe\") " Oct 30 20:06:34.508373 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:34.508246 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe-credential-socket" pod "cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe" (UID: "cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:06:34.508568 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:34.508517 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe-workload-certs" pod "cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe" (UID: "cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:06:34.508568 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:34.508527 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe-istiod-ca-cert" pod "cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe" (UID: "cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:06:34.508701 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:34.508627 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe-workload-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:34.508701 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:34.508649 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe-credential-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:34.508701 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:34.508662 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe-istiod-ca-cert\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:34.508701 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:34.508676 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe-workload-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:34.508924 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:34.508719 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe-istio-data" pod "cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe" (UID: "cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:06:34.510221 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:34.510198 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe-kube-api-access-plj6d" pod "cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe" (UID: "cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe"). InnerVolumeSpecName "kube-api-access-plj6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:06:34.510555 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:34.510527 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe-istio-envoy" pod "cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe" (UID: "cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:06:34.510555 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:34.510536 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe-istio-podinfo" pod "cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe" (UID: "cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:06:34.510716 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:34.510695 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe-istio-token" pod "cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe" (UID: "cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:06:34.597417 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:34.597390 4292 generic.go:356] "Generic (PLEG): container finished" podID="cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe" containerID="9f19d7f1745cd961b77bab3ee2f4be2bd854c951b726fdc08ff73058a6c196fd" exitCode=0 Oct 30 20:06:34.597544 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:34.597436 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr" Oct 30 20:06:34.597544 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:34.597474 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr" event={"ID":"cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe","Type":"ContainerDied","Data":"9f19d7f1745cd961b77bab3ee2f4be2bd854c951b726fdc08ff73058a6c196fd"} Oct 30 20:06:34.597544 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:34.597521 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr" event={"ID":"cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe","Type":"ContainerDied","Data":"9b60291c83f0c65cd4893573ba4ce9209e1e3fd1888c951fa6a2605f69a7760d"} Oct 30 20:06:34.597544 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:34.597543 4292 scope.go:122] "RemoveContainer" containerID="9f19d7f1745cd961b77bab3ee2f4be2bd854c951b726fdc08ff73058a6c196fd" Oct 30 20:06:34.609572 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:34.609556 4292 scope.go:122] "RemoveContainer" containerID="6b0c868cf678f8da347762ae591ea92b7ead5baa887c9be3d21cb5dc98c83c61" Oct 30 20:06:34.609721 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:34.609699 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe-istio-data\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:34.609787 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:34.609729 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe-istio-podinfo\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:34.609787 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:34.609745 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe-istio-token\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:34.609787 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:34.609759 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe-istio-envoy\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:34.609787 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:34.609775 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plj6d\" (UniqueName: \"kubernetes.io/projected/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe-kube-api-access-plj6d\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:34.621942 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:34.621921 4292 scope.go:122] "RemoveContainer" containerID="9f19d7f1745cd961b77bab3ee2f4be2bd854c951b726fdc08ff73058a6c196fd" Oct 30 20:06:34.622235 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:06:34.622216 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f19d7f1745cd961b77bab3ee2f4be2bd854c951b726fdc08ff73058a6c196fd\": container with ID starting with 9f19d7f1745cd961b77bab3ee2f4be2bd854c951b726fdc08ff73058a6c196fd not found: ID does not exist" containerID="9f19d7f1745cd961b77bab3ee2f4be2bd854c951b726fdc08ff73058a6c196fd" Oct 30 20:06:34.622292 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:34.622243 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f19d7f1745cd961b77bab3ee2f4be2bd854c951b726fdc08ff73058a6c196fd"} err="failed to get container status \"9f19d7f1745cd961b77bab3ee2f4be2bd854c951b726fdc08ff73058a6c196fd\": rpc error: code = NotFound desc = could not find container \"9f19d7f1745cd961b77bab3ee2f4be2bd854c951b726fdc08ff73058a6c196fd\": container with ID starting with 9f19d7f1745cd961b77bab3ee2f4be2bd854c951b726fdc08ff73058a6c196fd not found: ID does not exist" Oct 30 20:06:34.622292 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:34.622263 4292 scope.go:122] "RemoveContainer" containerID="6b0c868cf678f8da347762ae591ea92b7ead5baa887c9be3d21cb5dc98c83c61" Oct 30 20:06:34.622526 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:06:34.622505 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b0c868cf678f8da347762ae591ea92b7ead5baa887c9be3d21cb5dc98c83c61\": container with ID starting with 6b0c868cf678f8da347762ae591ea92b7ead5baa887c9be3d21cb5dc98c83c61 not found: ID does not exist" containerID="6b0c868cf678f8da347762ae591ea92b7ead5baa887c9be3d21cb5dc98c83c61" Oct 30 20:06:34.622571 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:34.622533 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b0c868cf678f8da347762ae591ea92b7ead5baa887c9be3d21cb5dc98c83c61"} err="failed to get container status \"6b0c868cf678f8da347762ae591ea92b7ead5baa887c9be3d21cb5dc98c83c61\": rpc error: code = NotFound desc = could not find container \"6b0c868cf678f8da347762ae591ea92b7ead5baa887c9be3d21cb5dc98c83c61\": container with ID starting with 6b0c868cf678f8da347762ae591ea92b7ead5baa887c9be3d21cb5dc98c83c61 not found: ID does not exist" Oct 30 20:06:34.769000 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:34.768967 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr"] Oct 30 20:06:34.773424 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:34.773396 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-8b0edc-openshift-default-7ccf7c4f7-bl5zr"] Oct 30 20:06:36.582668 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:06:36.582638 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe" path="/var/lib/kubelet/pods/cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe/volumes" Oct 30 20:06:43.221301 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:06:43.221260 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:07:07.719218 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:07.719176 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2d2ca9-76879b9f9-97sgs"] Oct 30 20:07:07.719749 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:07.719402 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d2ca9-76879b9f9-97sgs" podUID="cf9b7462-be83-4c4d-ba6f-5b8f86ab431b" containerName="payload-pre-processing" containerID="cri-o://18458718805e25d4812b94bbbb0dec04fdf1ca3e433e5568062998546a8f1075" gracePeriod=30 Oct 30 20:07:07.983660 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:07.983605 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d2ca9-76879b9f9-97sgs" Oct 30 20:07:08.129845 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:08.129811 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf9b7462-be83-4c4d-ba6f-5b8f86ab431b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf9b7462-be83-4c4d-ba6f-5b8f86ab431b-plugins-config-volume\") pod \"cf9b7462-be83-4c4d-ba6f-5b8f86ab431b\" (UID: \"cf9b7462-be83-4c4d-ba6f-5b8f86ab431b\") " Oct 30 20:07:08.130023 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:08.129874 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf9b7462-be83-4c4d-ba6f-5b8f86ab431b-kube-api-access-28jqj\" (UniqueName: \"kubernetes.io/projected/cf9b7462-be83-4c4d-ba6f-5b8f86ab431b-kube-api-access-28jqj\") pod \"cf9b7462-be83-4c4d-ba6f-5b8f86ab431b\" (UID: \"cf9b7462-be83-4c4d-ba6f-5b8f86ab431b\") " Oct 30 20:07:08.130318 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:08.130287 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf9b7462-be83-4c4d-ba6f-5b8f86ab431b-plugins-config-volume" pod "cf9b7462-be83-4c4d-ba6f-5b8f86ab431b" (UID: "cf9b7462-be83-4c4d-ba6f-5b8f86ab431b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:07:08.131988 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:08.131957 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf9b7462-be83-4c4d-ba6f-5b8f86ab431b-kube-api-access-28jqj" pod "cf9b7462-be83-4c4d-ba6f-5b8f86ab431b" (UID: "cf9b7462-be83-4c4d-ba6f-5b8f86ab431b"). InnerVolumeSpecName "kube-api-access-28jqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:07:08.237427 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:08.236030 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf9b7462-be83-4c4d-ba6f-5b8f86ab431b-plugins-config-volume\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:07:08.237427 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:08.236069 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28jqj\" (UniqueName: \"kubernetes.io/projected/cf9b7462-be83-4c4d-ba6f-5b8f86ab431b-kube-api-access-28jqj\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:07:08.805371 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:08.805042 4292 generic.go:356] "Generic (PLEG): container finished" podID="cf9b7462-be83-4c4d-ba6f-5b8f86ab431b" containerID="18458718805e25d4812b94bbbb0dec04fdf1ca3e433e5568062998546a8f1075" exitCode=0 Oct 30 20:07:08.805371 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:08.805162 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d2ca9-76879b9f9-97sgs" event={"ID":"cf9b7462-be83-4c4d-ba6f-5b8f86ab431b","Type":"ContainerDied","Data":"18458718805e25d4812b94bbbb0dec04fdf1ca3e433e5568062998546a8f1075"} Oct 30 20:07:08.805371 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:08.805213 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d2ca9-76879b9f9-97sgs" event={"ID":"cf9b7462-be83-4c4d-ba6f-5b8f86ab431b","Type":"ContainerDied","Data":"adf59c507dc50b92230214827d37d56b5d89eaeb212b8011a363dfb354fbe3d0"} Oct 30 20:07:08.805371 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:08.805244 4292 scope.go:122] "RemoveContainer" containerID="18458718805e25d4812b94bbbb0dec04fdf1ca3e433e5568062998546a8f1075" Oct 30 20:07:08.806034 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:08.805451 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d2ca9-76879b9f9-97sgs" Oct 30 20:07:08.820797 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:08.820772 4292 scope.go:122] "RemoveContainer" containerID="18458718805e25d4812b94bbbb0dec04fdf1ca3e433e5568062998546a8f1075" Oct 30 20:07:08.821140 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:07:08.821108 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18458718805e25d4812b94bbbb0dec04fdf1ca3e433e5568062998546a8f1075\": container with ID starting with 18458718805e25d4812b94bbbb0dec04fdf1ca3e433e5568062998546a8f1075 not found: ID does not exist" containerID="18458718805e25d4812b94bbbb0dec04fdf1ca3e433e5568062998546a8f1075" Oct 30 20:07:08.821243 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:08.821146 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18458718805e25d4812b94bbbb0dec04fdf1ca3e433e5568062998546a8f1075"} err="failed to get container status \"18458718805e25d4812b94bbbb0dec04fdf1ca3e433e5568062998546a8f1075\": rpc error: code = NotFound desc = could not find container \"18458718805e25d4812b94bbbb0dec04fdf1ca3e433e5568062998546a8f1075\": container with ID starting with 18458718805e25d4812b94bbbb0dec04fdf1ca3e433e5568062998546a8f1075 not found: ID does not exist" Oct 30 20:07:08.976650 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:08.976578 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2d2ca9-76879b9f9-97sgs"] Oct 30 20:07:09.027615 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:09.027585 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2d2ca9-76879b9f9-97sgs"] Oct 30 20:07:10.584221 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:10.584168 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf9b7462-be83-4c4d-ba6f-5b8f86ab431b" path="/var/lib/kubelet/pods/cf9b7462-be83-4c4d-ba6f-5b8f86ab431b/volumes" Oct 30 20:07:13.220826 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:07:13.220788 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:07:17.639285 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:17.639252 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9"] Oct 30 20:07:17.639707 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:17.639484 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9" podUID="61b24a37-2664-40ff-a96e-58f2d44e1045" containerName="istio-proxy" containerID="cri-o://9199cf36052feef81c8921548c3bf80f60d62b36c6f1e168967befd0bba1ebc7" gracePeriod=30 Oct 30 20:07:22.895911 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:22.895884 4292 generic.go:356] "Generic (PLEG): container finished" podID="61b24a37-2664-40ff-a96e-58f2d44e1045" containerID="9199cf36052feef81c8921548c3bf80f60d62b36c6f1e168967befd0bba1ebc7" exitCode=0 Oct 30 20:07:22.896255 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:22.895957 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9" event={"ID":"61b24a37-2664-40ff-a96e-58f2d44e1045","Type":"ContainerDied","Data":"9199cf36052feef81c8921548c3bf80f60d62b36c6f1e168967befd0bba1ebc7"} Oct 30 20:07:22.896255 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:22.896008 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9" event={"ID":"61b24a37-2664-40ff-a96e-58f2d44e1045","Type":"ContainerDied","Data":"f862dc2f7a363d68faa0fb1c599948de14b159ac883ba5fca71f1e114505eb11"} Oct 30 20:07:22.896255 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:22.896024 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f862dc2f7a363d68faa0fb1c599948de14b159ac883ba5fca71f1e114505eb11" Oct 30 20:07:22.896255 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:22.896037 4292 scope.go:122] "RemoveContainer" containerID="de7e331af7e302ee6ae20cd723fd13e3b790e95de0a254d4bba55c5c01a7b89b" Oct 30 20:07:22.896944 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:22.896892 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9" Oct 30 20:07:23.024566 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:23.024536 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61b24a37-2664-40ff-a96e-58f2d44e1045-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61b24a37-2664-40ff-a96e-58f2d44e1045-istiod-ca-cert\") pod \"61b24a37-2664-40ff-a96e-58f2d44e1045\" (UID: \"61b24a37-2664-40ff-a96e-58f2d44e1045\") " Oct 30 20:07:23.024735 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:23.024580 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61b24a37-2664-40ff-a96e-58f2d44e1045-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61b24a37-2664-40ff-a96e-58f2d44e1045-istio-data\") pod \"61b24a37-2664-40ff-a96e-58f2d44e1045\" (UID: \"61b24a37-2664-40ff-a96e-58f2d44e1045\") " Oct 30 20:07:23.024735 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:23.024630 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61b24a37-2664-40ff-a96e-58f2d44e1045-kube-api-access-tk5p9\" (UniqueName: \"kubernetes.io/projected/61b24a37-2664-40ff-a96e-58f2d44e1045-kube-api-access-tk5p9\") pod \"61b24a37-2664-40ff-a96e-58f2d44e1045\" (UID: \"61b24a37-2664-40ff-a96e-58f2d44e1045\") " Oct 30 20:07:23.024735 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:23.024677 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61b24a37-2664-40ff-a96e-58f2d44e1045-istio-token\" (UniqueName: \"kubernetes.io/projected/61b24a37-2664-40ff-a96e-58f2d44e1045-istio-token\") pod \"61b24a37-2664-40ff-a96e-58f2d44e1045\" (UID: \"61b24a37-2664-40ff-a96e-58f2d44e1045\") " Oct 30 20:07:23.024881 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:23.024734 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61b24a37-2664-40ff-a96e-58f2d44e1045-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61b24a37-2664-40ff-a96e-58f2d44e1045-credential-socket\") pod \"61b24a37-2664-40ff-a96e-58f2d44e1045\" (UID: \"61b24a37-2664-40ff-a96e-58f2d44e1045\") " Oct 30 20:07:23.024881 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:23.024761 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61b24a37-2664-40ff-a96e-58f2d44e1045-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61b24a37-2664-40ff-a96e-58f2d44e1045-workload-certs\") pod \"61b24a37-2664-40ff-a96e-58f2d44e1045\" (UID: \"61b24a37-2664-40ff-a96e-58f2d44e1045\") " Oct 30 20:07:23.024881 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:23.024795 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61b24a37-2664-40ff-a96e-58f2d44e1045-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61b24a37-2664-40ff-a96e-58f2d44e1045-workload-socket\") pod \"61b24a37-2664-40ff-a96e-58f2d44e1045\" (UID: \"61b24a37-2664-40ff-a96e-58f2d44e1045\") " Oct 30 20:07:23.024881 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:23.024824 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61b24a37-2664-40ff-a96e-58f2d44e1045-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61b24a37-2664-40ff-a96e-58f2d44e1045-istio-envoy\") pod \"61b24a37-2664-40ff-a96e-58f2d44e1045\" (UID: \"61b24a37-2664-40ff-a96e-58f2d44e1045\") " Oct 30 20:07:23.025081 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:23.024901 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61b24a37-2664-40ff-a96e-58f2d44e1045-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61b24a37-2664-40ff-a96e-58f2d44e1045-istio-podinfo\") pod \"61b24a37-2664-40ff-a96e-58f2d44e1045\" (UID: \"61b24a37-2664-40ff-a96e-58f2d44e1045\") " Oct 30 20:07:23.025081 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:23.024956 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61b24a37-2664-40ff-a96e-58f2d44e1045-istio-data" pod "61b24a37-2664-40ff-a96e-58f2d44e1045" (UID: "61b24a37-2664-40ff-a96e-58f2d44e1045"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:07:23.025081 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:23.024964 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61b24a37-2664-40ff-a96e-58f2d44e1045-istiod-ca-cert" pod "61b24a37-2664-40ff-a96e-58f2d44e1045" (UID: "61b24a37-2664-40ff-a96e-58f2d44e1045"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:07:23.025233 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:23.025075 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61b24a37-2664-40ff-a96e-58f2d44e1045-workload-certs" pod "61b24a37-2664-40ff-a96e-58f2d44e1045" (UID: "61b24a37-2664-40ff-a96e-58f2d44e1045"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:07:23.025233 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:23.025091 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61b24a37-2664-40ff-a96e-58f2d44e1045-credential-socket" pod "61b24a37-2664-40ff-a96e-58f2d44e1045" (UID: "61b24a37-2664-40ff-a96e-58f2d44e1045"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:07:23.025334 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:23.025249 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61b24a37-2664-40ff-a96e-58f2d44e1045-workload-socket" pod "61b24a37-2664-40ff-a96e-58f2d44e1045" (UID: "61b24a37-2664-40ff-a96e-58f2d44e1045"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:07:23.025457 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:23.025433 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61b24a37-2664-40ff-a96e-58f2d44e1045-credential-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:07:23.025457 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:23.025459 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61b24a37-2664-40ff-a96e-58f2d44e1045-workload-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:07:23.025656 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:23.025471 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61b24a37-2664-40ff-a96e-58f2d44e1045-workload-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:07:23.025656 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:23.025480 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61b24a37-2664-40ff-a96e-58f2d44e1045-istiod-ca-cert\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:07:23.025656 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:23.025488 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61b24a37-2664-40ff-a96e-58f2d44e1045-istio-data\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:07:23.026954 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:23.026930 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61b24a37-2664-40ff-a96e-58f2d44e1045-istio-token" pod "61b24a37-2664-40ff-a96e-58f2d44e1045" (UID: "61b24a37-2664-40ff-a96e-58f2d44e1045"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:07:23.027058 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:23.027040 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61b24a37-2664-40ff-a96e-58f2d44e1045-istio-podinfo" pod "61b24a37-2664-40ff-a96e-58f2d44e1045" (UID: "61b24a37-2664-40ff-a96e-58f2d44e1045"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:07:23.027261 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:23.027242 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61b24a37-2664-40ff-a96e-58f2d44e1045-istio-envoy" pod "61b24a37-2664-40ff-a96e-58f2d44e1045" (UID: "61b24a37-2664-40ff-a96e-58f2d44e1045"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:07:23.027319 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:23.027264 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61b24a37-2664-40ff-a96e-58f2d44e1045-kube-api-access-tk5p9" pod "61b24a37-2664-40ff-a96e-58f2d44e1045" (UID: "61b24a37-2664-40ff-a96e-58f2d44e1045"). InnerVolumeSpecName "kube-api-access-tk5p9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:07:23.126039 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:23.126009 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61b24a37-2664-40ff-a96e-58f2d44e1045-istio-envoy\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:07:23.126039 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:23.126037 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61b24a37-2664-40ff-a96e-58f2d44e1045-istio-podinfo\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:07:23.126240 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:23.126048 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tk5p9\" (UniqueName: \"kubernetes.io/projected/61b24a37-2664-40ff-a96e-58f2d44e1045-kube-api-access-tk5p9\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:07:23.126240 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:23.126057 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61b24a37-2664-40ff-a96e-58f2d44e1045-istio-token\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:07:23.902093 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:23.902066 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9" Oct 30 20:07:24.197475 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:24.197394 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9"] Oct 30 20:07:24.199511 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:24.199500 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-2d2ca9-openshift-default-7c5b8f9cd5-j45d9"] Oct 30 20:07:24.583242 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:24.583207 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61b24a37-2664-40ff-a96e-58f2d44e1045" path="/var/lib/kubelet/pods/61b24a37-2664-40ff-a96e-58f2d44e1045/volumes" Oct 30 20:07:30.156183 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:30.156151 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f700fe-56cffff755-ck8pz"] Oct 30 20:07:35.156704 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:35.156650 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f700fe-56cffff755-ck8pz" podUID="26b6bea7-3ad9-4c8d-85b9-fb2edca9afce" containerName="payload-pre-processing" containerID="cri-o://5bb4cc923f663d15def9c3bf751f03d8a903f9b9ea59660ae1c98be69bd4b984" gracePeriod=25 Oct 30 20:07:35.480674 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:35.480649 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f700fe-56cffff755-ck8pz" Oct 30 20:07:35.502648 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:35.502623 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26b6bea7-3ad9-4c8d-85b9-fb2edca9afce-kube-api-access-slssp\" (UniqueName: \"kubernetes.io/projected/26b6bea7-3ad9-4c8d-85b9-fb2edca9afce-kube-api-access-slssp\") pod \"26b6bea7-3ad9-4c8d-85b9-fb2edca9afce\" (UID: \"26b6bea7-3ad9-4c8d-85b9-fb2edca9afce\") " Oct 30 20:07:35.502793 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:35.502686 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26b6bea7-3ad9-4c8d-85b9-fb2edca9afce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26b6bea7-3ad9-4c8d-85b9-fb2edca9afce-plugins-config-volume\") pod \"26b6bea7-3ad9-4c8d-85b9-fb2edca9afce\" (UID: \"26b6bea7-3ad9-4c8d-85b9-fb2edca9afce\") " Oct 30 20:07:35.503088 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:35.503066 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26b6bea7-3ad9-4c8d-85b9-fb2edca9afce-plugins-config-volume" pod "26b6bea7-3ad9-4c8d-85b9-fb2edca9afce" (UID: "26b6bea7-3ad9-4c8d-85b9-fb2edca9afce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:07:35.504876 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:35.504850 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26b6bea7-3ad9-4c8d-85b9-fb2edca9afce-kube-api-access-slssp" pod "26b6bea7-3ad9-4c8d-85b9-fb2edca9afce" (UID: "26b6bea7-3ad9-4c8d-85b9-fb2edca9afce"). InnerVolumeSpecName "kube-api-access-slssp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:07:35.604069 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:35.604037 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slssp\" (UniqueName: \"kubernetes.io/projected/26b6bea7-3ad9-4c8d-85b9-fb2edca9afce-kube-api-access-slssp\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:07:35.604069 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:35.604068 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26b6bea7-3ad9-4c8d-85b9-fb2edca9afce-plugins-config-volume\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:07:35.966465 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:35.966434 4292 generic.go:356] "Generic (PLEG): container finished" podID="26b6bea7-3ad9-4c8d-85b9-fb2edca9afce" containerID="5bb4cc923f663d15def9c3bf751f03d8a903f9b9ea59660ae1c98be69bd4b984" exitCode=0 Oct 30 20:07:35.966628 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:35.966482 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f700fe-56cffff755-ck8pz" event={"ID":"26b6bea7-3ad9-4c8d-85b9-fb2edca9afce","Type":"ContainerDied","Data":"5bb4cc923f663d15def9c3bf751f03d8a903f9b9ea59660ae1c98be69bd4b984"} Oct 30 20:07:35.966628 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:35.966497 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f700fe-56cffff755-ck8pz" Oct 30 20:07:35.966628 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:35.966526 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f700fe-56cffff755-ck8pz" event={"ID":"26b6bea7-3ad9-4c8d-85b9-fb2edca9afce","Type":"ContainerDied","Data":"b89d017f16acfe9732713230d2571187227eb1bb288daa4a73b4c4e7649fe615"} Oct 30 20:07:35.966628 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:35.966547 4292 scope.go:122] "RemoveContainer" containerID="5bb4cc923f663d15def9c3bf751f03d8a903f9b9ea59660ae1c98be69bd4b984" Oct 30 20:07:35.975622 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:35.975598 4292 scope.go:122] "RemoveContainer" containerID="5bb4cc923f663d15def9c3bf751f03d8a903f9b9ea59660ae1c98be69bd4b984" Oct 30 20:07:35.975901 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:07:35.975874 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bb4cc923f663d15def9c3bf751f03d8a903f9b9ea59660ae1c98be69bd4b984\": container with ID starting with 5bb4cc923f663d15def9c3bf751f03d8a903f9b9ea59660ae1c98be69bd4b984 not found: ID does not exist" containerID="5bb4cc923f663d15def9c3bf751f03d8a903f9b9ea59660ae1c98be69bd4b984" Oct 30 20:07:35.975971 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:35.975915 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bb4cc923f663d15def9c3bf751f03d8a903f9b9ea59660ae1c98be69bd4b984"} err="failed to get container status \"5bb4cc923f663d15def9c3bf751f03d8a903f9b9ea59660ae1c98be69bd4b984\": rpc error: code = NotFound desc = could not find container \"5bb4cc923f663d15def9c3bf751f03d8a903f9b9ea59660ae1c98be69bd4b984\": container with ID starting with 5bb4cc923f663d15def9c3bf751f03d8a903f9b9ea59660ae1c98be69bd4b984 not found: ID does not exist" Oct 30 20:07:36.136079 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:36.136044 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f700fe-56cffff755-ck8pz"] Oct 30 20:07:36.140548 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:36.140522 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f700fe-56cffff755-ck8pz"] Oct 30 20:07:36.584129 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:07:36.584096 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26b6bea7-3ad9-4c8d-85b9-fb2edca9afce" path="/var/lib/kubelet/pods/26b6bea7-3ad9-4c8d-85b9-fb2edca9afce/volumes" Oct 30 20:07:43.221181 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:07:43.221089 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:08:13.222663 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:08:13.222619 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:08:32.520823 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.520782 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49"] Oct 30 20:08:32.521321 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.521225 4292 cpu_manager.go:414] "containerMap: removing container" podUID="554fd8be-cebc-4035-840e-e377169383e5" containerName="istio-proxy" Oct 30 20:08:32.521321 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.521236 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="554fd8be-cebc-4035-840e-e377169383e5" containerName="istio-proxy" Oct 30 20:08:32.521321 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.521247 4292 cpu_manager.go:414] "containerMap: removing container" podUID="26b6bea7-3ad9-4c8d-85b9-fb2edca9afce" containerName="payload-pre-processing" Oct 30 20:08:32.521321 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.521252 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26b6bea7-3ad9-4c8d-85b9-fb2edca9afce" containerName="payload-pre-processing" Oct 30 20:08:32.521321 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.521261 4292 cpu_manager.go:414] "containerMap: removing container" podUID="cf9b7462-be83-4c4d-ba6f-5b8f86ab431b" containerName="payload-pre-processing" Oct 30 20:08:32.521321 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.521266 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf9b7462-be83-4c4d-ba6f-5b8f86ab431b" containerName="payload-pre-processing" Oct 30 20:08:32.521321 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.521280 4292 cpu_manager.go:414] "containerMap: removing container" podUID="554fd8be-cebc-4035-840e-e377169383e5" containerName="istio-proxy" Oct 30 20:08:32.521321 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.521285 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="554fd8be-cebc-4035-840e-e377169383e5" containerName="istio-proxy" Oct 30 20:08:32.521321 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.521291 4292 cpu_manager.go:414] "containerMap: removing container" podUID="61b24a37-2664-40ff-a96e-58f2d44e1045" containerName="istio-proxy" Oct 30 20:08:32.521321 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.521296 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61b24a37-2664-40ff-a96e-58f2d44e1045" containerName="istio-proxy" Oct 30 20:08:32.521321 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.521316 4292 cpu_manager.go:414] "containerMap: removing container" podUID="cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe" containerName="istio-proxy" Oct 30 20:08:32.521321 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.521320 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe" containerName="istio-proxy" Oct 30 20:08:32.521321 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.521334 4292 cpu_manager.go:414] "containerMap: removing container" podUID="9ec677ce-8380-4793-b6cf-02a092f9ffa5" containerName="payload-pre-processing" Oct 30 20:08:32.521762 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.521339 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ec677ce-8380-4793-b6cf-02a092f9ffa5" containerName="payload-pre-processing" Oct 30 20:08:32.521762 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.521361 4292 cpu_manager.go:414] "containerMap: removing container" podUID="6211173e-2380-427d-9501-bd4708e62784" containerName="revoke-keys" Oct 30 20:08:32.521762 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.521366 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6211173e-2380-427d-9501-bd4708e62784" containerName="revoke-keys" Oct 30 20:08:32.521762 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.521372 4292 cpu_manager.go:414] "containerMap: removing container" podUID="cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe" containerName="istio-proxy" Oct 30 20:08:32.521762 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.521377 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe" containerName="istio-proxy" Oct 30 20:08:32.521762 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.521438 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="61b24a37-2664-40ff-a96e-58f2d44e1045" containerName="istio-proxy" Oct 30 20:08:32.521762 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.521445 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe" containerName="istio-proxy" Oct 30 20:08:32.521762 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.521452 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ec677ce-8380-4793-b6cf-02a092f9ffa5" containerName="payload-pre-processing" Oct 30 20:08:32.521762 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.521458 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="6211173e-2380-427d-9501-bd4708e62784" containerName="revoke-keys" Oct 30 20:08:32.521762 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.521464 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd3ad97f-a09f-4932-b5f5-b4a0a8640ebe" containerName="istio-proxy" Oct 30 20:08:32.521762 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.521470 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="554fd8be-cebc-4035-840e-e377169383e5" containerName="istio-proxy" Oct 30 20:08:32.521762 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.521475 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="26b6bea7-3ad9-4c8d-85b9-fb2edca9afce" containerName="payload-pre-processing" Oct 30 20:08:32.521762 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.521480 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf9b7462-be83-4c4d-ba6f-5b8f86ab431b" containerName="payload-pre-processing" Oct 30 20:08:32.521762 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.521486 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="554fd8be-cebc-4035-840e-e377169383e5" containerName="istio-proxy" Oct 30 20:08:32.524161 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.524145 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49" Oct 30 20:08:32.571306 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.571274 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd"] Oct 30 20:08:32.571924 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.571900 4292 cpu_manager.go:414] "containerMap: removing container" podUID="61b24a37-2664-40ff-a96e-58f2d44e1045" containerName="istio-proxy" Oct 30 20:08:32.572095 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.572069 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61b24a37-2664-40ff-a96e-58f2d44e1045" containerName="istio-proxy" Oct 30 20:08:32.572274 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.572256 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="61b24a37-2664-40ff-a96e-58f2d44e1045" containerName="istio-proxy" Oct 30 20:08:32.576854 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.576828 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd" Oct 30 20:08:32.586465 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.586437 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed835dd4-48ed-46cf-9180-d616d5fcec23-istio-envoy\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49\" (UID: \"ed835dd4-48ed-46cf-9180-d616d5fcec23\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49" Oct 30 20:08:32.586617 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.586478 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed835dd4-48ed-46cf-9180-d616d5fcec23-workload-certs\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49\" (UID: \"ed835dd4-48ed-46cf-9180-d616d5fcec23\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49" Oct 30 20:08:32.586617 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.586517 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed835dd4-48ed-46cf-9180-d616d5fcec23-workload-socket\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49\" (UID: \"ed835dd4-48ed-46cf-9180-d616d5fcec23\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49" Oct 30 20:08:32.586617 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.586573 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed835dd4-48ed-46cf-9180-d616d5fcec23-credential-socket\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49\" (UID: \"ed835dd4-48ed-46cf-9180-d616d5fcec23\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49" Oct 30 20:08:32.586763 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.586640 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62td5\" (UniqueName: \"kubernetes.io/projected/ed835dd4-48ed-46cf-9180-d616d5fcec23-kube-api-access-62td5\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49\" (UID: \"ed835dd4-48ed-46cf-9180-d616d5fcec23\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49" Oct 30 20:08:32.586763 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.586681 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed835dd4-48ed-46cf-9180-d616d5fcec23-istio-podinfo\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49\" (UID: \"ed835dd4-48ed-46cf-9180-d616d5fcec23\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49" Oct 30 20:08:32.586763 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.586708 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed835dd4-48ed-46cf-9180-d616d5fcec23-istio-token\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49\" (UID: \"ed835dd4-48ed-46cf-9180-d616d5fcec23\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49" Oct 30 20:08:32.586920 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.586774 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed835dd4-48ed-46cf-9180-d616d5fcec23-istio-data\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49\" (UID: \"ed835dd4-48ed-46cf-9180-d616d5fcec23\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49" Oct 30 20:08:32.586920 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.586813 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed835dd4-48ed-46cf-9180-d616d5fcec23-istiod-ca-cert\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49\" (UID: \"ed835dd4-48ed-46cf-9180-d616d5fcec23\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49" Oct 30 20:08:32.668368 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.657818 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-389c4e-openshift-default-dockercfg-kmrqw\"" Oct 30 20:08:32.668620 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.663431 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49"] Oct 30 20:08:32.668787 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.668770 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd"] Oct 30 20:08:32.699633 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.699600 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed835dd4-48ed-46cf-9180-d616d5fcec23-istio-data\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49\" (UID: \"ed835dd4-48ed-46cf-9180-d616d5fcec23\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49" Oct 30 20:08:32.699859 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.699840 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtclw\" (UniqueName: \"kubernetes.io/projected/095f29a0-8906-4da7-89ab-95fa4f9dc1a7-kube-api-access-xtclw\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd\" (UID: \"095f29a0-8906-4da7-89ab-95fa4f9dc1a7\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd" Oct 30 20:08:32.699995 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.699978 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed835dd4-48ed-46cf-9180-d616d5fcec23-istiod-ca-cert\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49\" (UID: \"ed835dd4-48ed-46cf-9180-d616d5fcec23\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49" Oct 30 20:08:32.700105 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.700091 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/095f29a0-8906-4da7-89ab-95fa4f9dc1a7-istio-podinfo\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd\" (UID: \"095f29a0-8906-4da7-89ab-95fa4f9dc1a7\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd" Oct 30 20:08:32.700204 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.700190 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/095f29a0-8906-4da7-89ab-95fa4f9dc1a7-credential-socket\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd\" (UID: \"095f29a0-8906-4da7-89ab-95fa4f9dc1a7\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd" Oct 30 20:08:32.700319 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.700307 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/095f29a0-8906-4da7-89ab-95fa4f9dc1a7-istio-token\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd\" (UID: \"095f29a0-8906-4da7-89ab-95fa4f9dc1a7\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd" Oct 30 20:08:32.700468 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.700452 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed835dd4-48ed-46cf-9180-d616d5fcec23-istio-envoy\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49\" (UID: \"ed835dd4-48ed-46cf-9180-d616d5fcec23\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49" Oct 30 20:08:32.700573 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.700559 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed835dd4-48ed-46cf-9180-d616d5fcec23-workload-certs\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49\" (UID: \"ed835dd4-48ed-46cf-9180-d616d5fcec23\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49" Oct 30 20:08:32.700668 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.700654 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/095f29a0-8906-4da7-89ab-95fa4f9dc1a7-workload-socket\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd\" (UID: \"095f29a0-8906-4da7-89ab-95fa4f9dc1a7\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd" Oct 30 20:08:32.700759 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.700748 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed835dd4-48ed-46cf-9180-d616d5fcec23-workload-socket\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49\" (UID: \"ed835dd4-48ed-46cf-9180-d616d5fcec23\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49" Oct 30 20:08:32.700946 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.700933 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed835dd4-48ed-46cf-9180-d616d5fcec23-credential-socket\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49\" (UID: \"ed835dd4-48ed-46cf-9180-d616d5fcec23\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49" Oct 30 20:08:32.701054 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.701042 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/095f29a0-8906-4da7-89ab-95fa4f9dc1a7-istio-envoy\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd\" (UID: \"095f29a0-8906-4da7-89ab-95fa4f9dc1a7\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd" Oct 30 20:08:32.701181 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.701168 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/095f29a0-8906-4da7-89ab-95fa4f9dc1a7-istiod-ca-cert\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd\" (UID: \"095f29a0-8906-4da7-89ab-95fa4f9dc1a7\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd" Oct 30 20:08:32.701275 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.701262 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62td5\" (UniqueName: \"kubernetes.io/projected/ed835dd4-48ed-46cf-9180-d616d5fcec23-kube-api-access-62td5\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49\" (UID: \"ed835dd4-48ed-46cf-9180-d616d5fcec23\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49" Oct 30 20:08:32.701430 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.701414 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed835dd4-48ed-46cf-9180-d616d5fcec23-istio-podinfo\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49\" (UID: \"ed835dd4-48ed-46cf-9180-d616d5fcec23\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49" Oct 30 20:08:32.701543 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.701531 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed835dd4-48ed-46cf-9180-d616d5fcec23-istio-token\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49\" (UID: \"ed835dd4-48ed-46cf-9180-d616d5fcec23\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49" Oct 30 20:08:32.701636 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.701623 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/095f29a0-8906-4da7-89ab-95fa4f9dc1a7-workload-certs\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd\" (UID: \"095f29a0-8906-4da7-89ab-95fa4f9dc1a7\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd" Oct 30 20:08:32.701739 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.701728 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/095f29a0-8906-4da7-89ab-95fa4f9dc1a7-istio-data\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd\" (UID: \"095f29a0-8906-4da7-89ab-95fa4f9dc1a7\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd" Oct 30 20:08:32.702615 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.702480 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed835dd4-48ed-46cf-9180-d616d5fcec23-istio-data\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49\" (UID: \"ed835dd4-48ed-46cf-9180-d616d5fcec23\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49" Oct 30 20:08:32.703344 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.703322 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed835dd4-48ed-46cf-9180-d616d5fcec23-istiod-ca-cert\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49\" (UID: \"ed835dd4-48ed-46cf-9180-d616d5fcec23\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49" Oct 30 20:08:32.705988 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.705966 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed835dd4-48ed-46cf-9180-d616d5fcec23-istio-envoy\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49\" (UID: \"ed835dd4-48ed-46cf-9180-d616d5fcec23\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49" Oct 30 20:08:32.706299 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.706277 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed835dd4-48ed-46cf-9180-d616d5fcec23-workload-certs\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49\" (UID: \"ed835dd4-48ed-46cf-9180-d616d5fcec23\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49" Oct 30 20:08:32.706548 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.706529 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed835dd4-48ed-46cf-9180-d616d5fcec23-workload-socket\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49\" (UID: \"ed835dd4-48ed-46cf-9180-d616d5fcec23\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49" Oct 30 20:08:32.707890 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.707380 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed835dd4-48ed-46cf-9180-d616d5fcec23-credential-socket\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49\" (UID: \"ed835dd4-48ed-46cf-9180-d616d5fcec23\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49" Oct 30 20:08:32.710947 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.710400 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed835dd4-48ed-46cf-9180-d616d5fcec23-istio-podinfo\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49\" (UID: \"ed835dd4-48ed-46cf-9180-d616d5fcec23\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49" Oct 30 20:08:32.777964 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.777885 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed835dd4-48ed-46cf-9180-d616d5fcec23-istio-token\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49\" (UID: \"ed835dd4-48ed-46cf-9180-d616d5fcec23\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49" Oct 30 20:08:32.778551 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.778529 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62td5\" (UniqueName: \"kubernetes.io/projected/ed835dd4-48ed-46cf-9180-d616d5fcec23-kube-api-access-62td5\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49\" (UID: \"ed835dd4-48ed-46cf-9180-d616d5fcec23\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49" Oct 30 20:08:32.802788 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.802751 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/095f29a0-8906-4da7-89ab-95fa4f9dc1a7-workload-certs\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd\" (UID: \"095f29a0-8906-4da7-89ab-95fa4f9dc1a7\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd" Oct 30 20:08:32.802933 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.802806 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/095f29a0-8906-4da7-89ab-95fa4f9dc1a7-istio-data\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd\" (UID: \"095f29a0-8906-4da7-89ab-95fa4f9dc1a7\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd" Oct 30 20:08:32.802933 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.802869 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtclw\" (UniqueName: \"kubernetes.io/projected/095f29a0-8906-4da7-89ab-95fa4f9dc1a7-kube-api-access-xtclw\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd\" (UID: \"095f29a0-8906-4da7-89ab-95fa4f9dc1a7\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd" Oct 30 20:08:32.802933 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.802924 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/095f29a0-8906-4da7-89ab-95fa4f9dc1a7-istio-podinfo\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd\" (UID: \"095f29a0-8906-4da7-89ab-95fa4f9dc1a7\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd" Oct 30 20:08:32.803088 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.802966 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/095f29a0-8906-4da7-89ab-95fa4f9dc1a7-credential-socket\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd\" (UID: \"095f29a0-8906-4da7-89ab-95fa4f9dc1a7\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd" Oct 30 20:08:32.803088 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.803020 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/095f29a0-8906-4da7-89ab-95fa4f9dc1a7-istio-token\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd\" (UID: \"095f29a0-8906-4da7-89ab-95fa4f9dc1a7\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd" Oct 30 20:08:32.803088 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.803063 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/095f29a0-8906-4da7-89ab-95fa4f9dc1a7-workload-socket\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd\" (UID: \"095f29a0-8906-4da7-89ab-95fa4f9dc1a7\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd" Oct 30 20:08:32.803247 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.803144 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/095f29a0-8906-4da7-89ab-95fa4f9dc1a7-istio-envoy\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd\" (UID: \"095f29a0-8906-4da7-89ab-95fa4f9dc1a7\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd" Oct 30 20:08:32.803247 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.803160 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/095f29a0-8906-4da7-89ab-95fa4f9dc1a7-workload-certs\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd\" (UID: \"095f29a0-8906-4da7-89ab-95fa4f9dc1a7\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd" Oct 30 20:08:32.803247 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.803195 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/095f29a0-8906-4da7-89ab-95fa4f9dc1a7-istiod-ca-cert\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd\" (UID: \"095f29a0-8906-4da7-89ab-95fa4f9dc1a7\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd" Oct 30 20:08:32.803247 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.803221 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/095f29a0-8906-4da7-89ab-95fa4f9dc1a7-istio-data\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd\" (UID: \"095f29a0-8906-4da7-89ab-95fa4f9dc1a7\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd" Oct 30 20:08:32.803652 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.803617 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/095f29a0-8906-4da7-89ab-95fa4f9dc1a7-credential-socket\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd\" (UID: \"095f29a0-8906-4da7-89ab-95fa4f9dc1a7\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd" Oct 30 20:08:32.803789 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.803654 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/095f29a0-8906-4da7-89ab-95fa4f9dc1a7-workload-socket\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd\" (UID: \"095f29a0-8906-4da7-89ab-95fa4f9dc1a7\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd" Oct 30 20:08:32.803906 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.803882 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/095f29a0-8906-4da7-89ab-95fa4f9dc1a7-istiod-ca-cert\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd\" (UID: \"095f29a0-8906-4da7-89ab-95fa4f9dc1a7\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd" Oct 30 20:08:32.805401 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.805381 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/095f29a0-8906-4da7-89ab-95fa4f9dc1a7-istio-envoy\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd\" (UID: \"095f29a0-8906-4da7-89ab-95fa4f9dc1a7\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd" Oct 30 20:08:32.805786 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.805763 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/095f29a0-8906-4da7-89ab-95fa4f9dc1a7-istio-podinfo\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd\" (UID: \"095f29a0-8906-4da7-89ab-95fa4f9dc1a7\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd" Oct 30 20:08:32.838376 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.838228 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49" Oct 30 20:08:32.865915 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.865886 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtclw\" (UniqueName: \"kubernetes.io/projected/095f29a0-8906-4da7-89ab-95fa4f9dc1a7-kube-api-access-xtclw\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd\" (UID: \"095f29a0-8906-4da7-89ab-95fa4f9dc1a7\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd" Oct 30 20:08:32.868797 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.868767 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/095f29a0-8906-4da7-89ab-95fa4f9dc1a7-istio-token\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd\" (UID: \"095f29a0-8906-4da7-89ab-95fa4f9dc1a7\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd" Oct 30 20:08:32.905275 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:32.905059 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd" Oct 30 20:08:33.307052 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:33.307015 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49"] Oct 30 20:08:33.307261 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:08:33.307242 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded835dd4_48ed_46cf_9180_d616d5fcec23.slice/crio-3521e7579601511a0079c74d02cba5ff38167c4fa87fc0f9728a44be60a75a34 WatchSource:0}: Error finding container 3521e7579601511a0079c74d02cba5ff38167c4fa87fc0f9728a44be60a75a34: Status 404 returned error can't find the container with id 3521e7579601511a0079c74d02cba5ff38167c4fa87fc0f9728a44be60a75a34 Oct 30 20:08:33.309972 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:33.309939 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:08:33.310091 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:33.310015 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:08:33.310091 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:33.310059 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:08:33.374630 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:08:33.374600 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod095f29a0_8906_4da7_89ab_95fa4f9dc1a7.slice/crio-795b115ea18d749f72842a3b3a4a77eb525d3f6c71679674cba49468d59c5e2e WatchSource:0}: Error finding container 795b115ea18d749f72842a3b3a4a77eb525d3f6c71679674cba49468d59c5e2e: Status 404 returned error can't find the container with id 795b115ea18d749f72842a3b3a4a77eb525d3f6c71679674cba49468d59c5e2e Oct 30 20:08:33.379202 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:33.378620 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd"] Oct 30 20:08:33.386388 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:33.386309 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:08:33.386646 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:33.386618 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:08:33.386840 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:33.386818 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:08:34.317009 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:34.316966 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd" event={"ID":"095f29a0-8906-4da7-89ab-95fa4f9dc1a7","Type":"ContainerStarted","Data":"62b7b5d5dc6af96b9cb548456d4ae9c59a0e27ddf2e47a8a86e82708ec218cdd"} Oct 30 20:08:34.317591 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:34.317035 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd" event={"ID":"095f29a0-8906-4da7-89ab-95fa4f9dc1a7","Type":"ContainerStarted","Data":"795b115ea18d749f72842a3b3a4a77eb525d3f6c71679674cba49468d59c5e2e"} Oct 30 20:08:34.318821 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:34.318796 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49" event={"ID":"ed835dd4-48ed-46cf-9180-d616d5fcec23","Type":"ContainerStarted","Data":"db8d85f2e8d793cc7e5b2414cb52537961457cab561346fea480117faf26da92"} Oct 30 20:08:34.318929 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:34.318838 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49" event={"ID":"ed835dd4-48ed-46cf-9180-d616d5fcec23","Type":"ContainerStarted","Data":"3521e7579601511a0079c74d02cba5ff38167c4fa87fc0f9728a44be60a75a34"} Oct 30 20:08:34.500140 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:34.500064 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd" podStartSLOduration=2.500044049 podStartE2EDuration="2.500044049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:08:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:08:34.498452668 +0000 UTC m=+2618.016768126" watchObservedRunningTime="2026-10-30 20:08:34.500044049 +0000 UTC m=+2618.018359508" Oct 30 20:08:34.642162 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:34.642029 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49" podStartSLOduration=2.642008089 podStartE2EDuration="2.642008089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:08:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:08:34.640198887 +0000 UTC m=+2618.158514345" watchObservedRunningTime="2026-10-30 20:08:34.642008089 +0000 UTC m=+2618.160323547" Oct 30 20:08:34.839370 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:34.839324 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49" Oct 30 20:08:34.840714 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:34.840684 4292 patch_prober.go:29] interesting pod/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 30 20:08:34.840765 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:34.840745 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49" podUID="ed835dd4-48ed-46cf-9180-d616d5fcec23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 30 20:08:34.908832 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:34.908740 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd" Oct 30 20:08:34.910175 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:34.910150 4292 patch_prober.go:29] interesting pod/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 30 20:08:34.910246 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:34.910200 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd" podUID="095f29a0-8906-4da7-89ab-95fa4f9dc1a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 30 20:08:35.838964 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:35.838928 4292 patch_prober.go:29] interesting pod/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 30 20:08:35.839374 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:35.838992 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49" podUID="ed835dd4-48ed-46cf-9180-d616d5fcec23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 30 20:08:35.906374 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:35.906038 4292 patch_prober.go:29] interesting pod/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 30 20:08:35.906374 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:35.906099 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd" podUID="095f29a0-8906-4da7-89ab-95fa4f9dc1a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 30 20:08:36.839373 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:36.839328 4292 patch_prober.go:29] interesting pod/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 30 20:08:36.839848 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:36.839414 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49" podUID="ed835dd4-48ed-46cf-9180-d616d5fcec23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 30 20:08:36.906095 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:36.906058 4292 patch_prober.go:29] interesting pod/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 30 20:08:36.906261 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:36.906109 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd" podUID="095f29a0-8906-4da7-89ab-95fa4f9dc1a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 30 20:08:37.839539 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:37.839502 4292 patch_prober.go:29] interesting pod/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 30 20:08:37.839933 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:37.839562 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49" podUID="ed835dd4-48ed-46cf-9180-d616d5fcec23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 30 20:08:37.906033 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:37.906001 4292 patch_prober.go:29] interesting pod/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 30 20:08:37.906263 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:37.906238 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd" podUID="095f29a0-8906-4da7-89ab-95fa4f9dc1a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 30 20:08:38.839147 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:38.839107 4292 patch_prober.go:29] interesting pod/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 30 20:08:38.839324 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:38.839178 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49" podUID="ed835dd4-48ed-46cf-9180-d616d5fcec23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 30 20:08:38.906610 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:38.906573 4292 patch_prober.go:29] interesting pod/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 30 20:08:38.906995 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:38.906648 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd" podUID="095f29a0-8906-4da7-89ab-95fa4f9dc1a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 30 20:08:39.843900 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:39.843871 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49" Oct 30 20:08:39.844159 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:39.844134 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49" Oct 30 20:08:39.844785 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:39.844767 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49" Oct 30 20:08:39.909971 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:39.909946 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd" Oct 30 20:08:39.910346 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:39.910195 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd" Oct 30 20:08:39.910877 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:39.910848 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd" Oct 30 20:08:43.220813 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:08:43.220777 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:08:44.704200 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:44.704160 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lfxgm"] Oct 30 20:08:44.704726 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:44.704434 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lfxgm" podUID="ac6a02b1-f1ae-4bce-a1ac-b925820fa432" containerName="istio-proxy" containerID="cri-o://94bce6f54a19ee4acc178c036b1e49d5a72ddc3c50be252d911e0fab61038846" gracePeriod=30 Oct 30 20:08:49.284012 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:49.283970 4292 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-lfxgm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:08:49.284394 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:49.284028 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lfxgm" podUID="ac6a02b1-f1ae-4bce-a1ac-b925820fa432" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:08:50.048179 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:50.048151 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lfxgm" Oct 30 20:08:50.078959 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:50.078926 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac6a02b1-f1ae-4bce-a1ac-b925820fa432-kube-api-access-h8xws\" (UniqueName: \"kubernetes.io/projected/ac6a02b1-f1ae-4bce-a1ac-b925820fa432-kube-api-access-h8xws\") pod \"ac6a02b1-f1ae-4bce-a1ac-b925820fa432\" (UID: \"ac6a02b1-f1ae-4bce-a1ac-b925820fa432\") " Oct 30 20:08:50.078959 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:50.078971 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac6a02b1-f1ae-4bce-a1ac-b925820fa432-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac6a02b1-f1ae-4bce-a1ac-b925820fa432-istio-podinfo\") pod \"ac6a02b1-f1ae-4bce-a1ac-b925820fa432\" (UID: \"ac6a02b1-f1ae-4bce-a1ac-b925820fa432\") " Oct 30 20:08:50.079217 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:50.079035 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac6a02b1-f1ae-4bce-a1ac-b925820fa432-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac6a02b1-f1ae-4bce-a1ac-b925820fa432-credential-socket\") pod \"ac6a02b1-f1ae-4bce-a1ac-b925820fa432\" (UID: \"ac6a02b1-f1ae-4bce-a1ac-b925820fa432\") " Oct 30 20:08:50.079217 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:50.079098 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac6a02b1-f1ae-4bce-a1ac-b925820fa432-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac6a02b1-f1ae-4bce-a1ac-b925820fa432-istio-data\") pod \"ac6a02b1-f1ae-4bce-a1ac-b925820fa432\" (UID: \"ac6a02b1-f1ae-4bce-a1ac-b925820fa432\") " Oct 30 20:08:50.079217 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:50.079199 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac6a02b1-f1ae-4bce-a1ac-b925820fa432-istio-token\" (UniqueName: \"kubernetes.io/projected/ac6a02b1-f1ae-4bce-a1ac-b925820fa432-istio-token\") pod \"ac6a02b1-f1ae-4bce-a1ac-b925820fa432\" (UID: \"ac6a02b1-f1ae-4bce-a1ac-b925820fa432\") " Oct 30 20:08:50.079397 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:50.079227 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac6a02b1-f1ae-4bce-a1ac-b925820fa432-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac6a02b1-f1ae-4bce-a1ac-b925820fa432-workload-socket\") pod \"ac6a02b1-f1ae-4bce-a1ac-b925820fa432\" (UID: \"ac6a02b1-f1ae-4bce-a1ac-b925820fa432\") " Oct 30 20:08:50.079397 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:50.079282 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac6a02b1-f1ae-4bce-a1ac-b925820fa432-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac6a02b1-f1ae-4bce-a1ac-b925820fa432-workload-certs\") pod \"ac6a02b1-f1ae-4bce-a1ac-b925820fa432\" (UID: \"ac6a02b1-f1ae-4bce-a1ac-b925820fa432\") " Oct 30 20:08:50.079397 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:50.079312 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac6a02b1-f1ae-4bce-a1ac-b925820fa432-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac6a02b1-f1ae-4bce-a1ac-b925820fa432-istio-envoy\") pod \"ac6a02b1-f1ae-4bce-a1ac-b925820fa432\" (UID: \"ac6a02b1-f1ae-4bce-a1ac-b925820fa432\") " Oct 30 20:08:50.079397 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:50.079369 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac6a02b1-f1ae-4bce-a1ac-b925820fa432-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac6a02b1-f1ae-4bce-a1ac-b925820fa432-istiod-ca-cert\") pod \"ac6a02b1-f1ae-4bce-a1ac-b925820fa432\" (UID: \"ac6a02b1-f1ae-4bce-a1ac-b925820fa432\") " Oct 30 20:08:50.080072 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:50.079989 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac6a02b1-f1ae-4bce-a1ac-b925820fa432-istio-data" pod "ac6a02b1-f1ae-4bce-a1ac-b925820fa432" (UID: "ac6a02b1-f1ae-4bce-a1ac-b925820fa432"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:50.080213 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:50.080128 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac6a02b1-f1ae-4bce-a1ac-b925820fa432-workload-socket" pod "ac6a02b1-f1ae-4bce-a1ac-b925820fa432" (UID: "ac6a02b1-f1ae-4bce-a1ac-b925820fa432"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:50.080602 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:50.080321 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac6a02b1-f1ae-4bce-a1ac-b925820fa432-workload-certs" pod "ac6a02b1-f1ae-4bce-a1ac-b925820fa432" (UID: "ac6a02b1-f1ae-4bce-a1ac-b925820fa432"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:50.080602 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:50.080397 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac6a02b1-f1ae-4bce-a1ac-b925820fa432-istiod-ca-cert" pod "ac6a02b1-f1ae-4bce-a1ac-b925820fa432" (UID: "ac6a02b1-f1ae-4bce-a1ac-b925820fa432"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:08:50.080889 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:50.080843 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac6a02b1-f1ae-4bce-a1ac-b925820fa432-credential-socket" pod "ac6a02b1-f1ae-4bce-a1ac-b925820fa432" (UID: "ac6a02b1-f1ae-4bce-a1ac-b925820fa432"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:50.081709 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:50.081673 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac6a02b1-f1ae-4bce-a1ac-b925820fa432-kube-api-access-h8xws" pod "ac6a02b1-f1ae-4bce-a1ac-b925820fa432" (UID: "ac6a02b1-f1ae-4bce-a1ac-b925820fa432"). InnerVolumeSpecName "kube-api-access-h8xws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:50.081818 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:50.081799 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac6a02b1-f1ae-4bce-a1ac-b925820fa432-istio-token" pod "ac6a02b1-f1ae-4bce-a1ac-b925820fa432" (UID: "ac6a02b1-f1ae-4bce-a1ac-b925820fa432"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:50.082156 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:50.082128 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac6a02b1-f1ae-4bce-a1ac-b925820fa432-istio-podinfo" pod "ac6a02b1-f1ae-4bce-a1ac-b925820fa432" (UID: "ac6a02b1-f1ae-4bce-a1ac-b925820fa432"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:08:50.082575 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:50.082556 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac6a02b1-f1ae-4bce-a1ac-b925820fa432-istio-envoy" pod "ac6a02b1-f1ae-4bce-a1ac-b925820fa432" (UID: "ac6a02b1-f1ae-4bce-a1ac-b925820fa432"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:08:50.181265 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:50.181229 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac6a02b1-f1ae-4bce-a1ac-b925820fa432-credential-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:08:50.181265 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:50.181268 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac6a02b1-f1ae-4bce-a1ac-b925820fa432-istio-data\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:08:50.181486 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:50.181283 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac6a02b1-f1ae-4bce-a1ac-b925820fa432-istio-token\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:08:50.181486 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:50.181297 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac6a02b1-f1ae-4bce-a1ac-b925820fa432-workload-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:08:50.181486 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:50.181313 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac6a02b1-f1ae-4bce-a1ac-b925820fa432-workload-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:08:50.181486 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:50.181334 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac6a02b1-f1ae-4bce-a1ac-b925820fa432-istio-envoy\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:08:50.181486 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:50.181381 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac6a02b1-f1ae-4bce-a1ac-b925820fa432-istiod-ca-cert\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:08:50.181486 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:50.181396 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8xws\" (UniqueName: \"kubernetes.io/projected/ac6a02b1-f1ae-4bce-a1ac-b925820fa432-kube-api-access-h8xws\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:08:50.181486 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:50.181410 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac6a02b1-f1ae-4bce-a1ac-b925820fa432-istio-podinfo\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:08:50.408160 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:50.408072 4292 generic.go:356] "Generic (PLEG): container finished" podID="ac6a02b1-f1ae-4bce-a1ac-b925820fa432" containerID="94bce6f54a19ee4acc178c036b1e49d5a72ddc3c50be252d911e0fab61038846" exitCode=0 Oct 30 20:08:50.408160 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:50.408135 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lfxgm" event={"ID":"ac6a02b1-f1ae-4bce-a1ac-b925820fa432","Type":"ContainerDied","Data":"94bce6f54a19ee4acc178c036b1e49d5a72ddc3c50be252d911e0fab61038846"} Oct 30 20:08:50.408862 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:50.408191 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lfxgm" event={"ID":"ac6a02b1-f1ae-4bce-a1ac-b925820fa432","Type":"ContainerDied","Data":"5fe8a2a4e479e429049d4c15399bf19002e526ead7b559f1049d2e000c5d8f1c"} Oct 30 20:08:50.408862 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:50.408204 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lfxgm" Oct 30 20:08:50.408862 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:50.408214 4292 scope.go:122] "RemoveContainer" containerID="94bce6f54a19ee4acc178c036b1e49d5a72ddc3c50be252d911e0fab61038846" Oct 30 20:08:50.418500 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:50.418475 4292 scope.go:122] "RemoveContainer" containerID="94bce6f54a19ee4acc178c036b1e49d5a72ddc3c50be252d911e0fab61038846" Oct 30 20:08:50.418791 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:08:50.418770 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94bce6f54a19ee4acc178c036b1e49d5a72ddc3c50be252d911e0fab61038846\": container with ID starting with 94bce6f54a19ee4acc178c036b1e49d5a72ddc3c50be252d911e0fab61038846 not found: ID does not exist" containerID="94bce6f54a19ee4acc178c036b1e49d5a72ddc3c50be252d911e0fab61038846" Oct 30 20:08:50.418870 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:50.418801 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94bce6f54a19ee4acc178c036b1e49d5a72ddc3c50be252d911e0fab61038846"} err="failed to get container status \"94bce6f54a19ee4acc178c036b1e49d5a72ddc3c50be252d911e0fab61038846\": rpc error: code = NotFound desc = could not find container \"94bce6f54a19ee4acc178c036b1e49d5a72ddc3c50be252d911e0fab61038846\": container with ID starting with 94bce6f54a19ee4acc178c036b1e49d5a72ddc3c50be252d911e0fab61038846 not found: ID does not exist" Oct 30 20:08:50.583669 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:50.583631 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lfxgm"] Oct 30 20:08:50.630761 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:50.630718 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lfxgm"] Oct 30 20:08:52.582549 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:52.582513 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac6a02b1-f1ae-4bce-a1ac-b925820fa432" path="/var/lib/kubelet/pods/ac6a02b1-f1ae-4bce-a1ac-b925820fa432/volumes" Oct 30 20:08:56.006653 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:56.006618 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp"] Oct 30 20:08:56.007090 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:56.007077 4292 cpu_manager.go:414] "containerMap: removing container" podUID="ac6a02b1-f1ae-4bce-a1ac-b925820fa432" containerName="istio-proxy" Oct 30 20:08:56.007135 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:56.007091 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac6a02b1-f1ae-4bce-a1ac-b925820fa432" containerName="istio-proxy" Oct 30 20:08:56.007174 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:56.007164 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac6a02b1-f1ae-4bce-a1ac-b925820fa432" containerName="istio-proxy" Oct 30 20:08:56.011759 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:56.011737 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp" Oct 30 20:08:56.067435 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:56.067409 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-f53eb9-openshift-default-dockercfg-2ldbv\"" Oct 30 20:08:56.084822 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:56.084777 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9543233d-9115-4459-87b7-dea47aa8d1ca-istio-podinfo\") pod \"e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp\" (UID: \"9543233d-9115-4459-87b7-dea47aa8d1ca\") " pod="openshift-ingress/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp" Oct 30 20:08:56.084822 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:56.084816 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9543233d-9115-4459-87b7-dea47aa8d1ca-istiod-ca-cert\") pod \"e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp\" (UID: \"9543233d-9115-4459-87b7-dea47aa8d1ca\") " pod="openshift-ingress/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp" Oct 30 20:08:56.085076 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:56.084902 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9543233d-9115-4459-87b7-dea47aa8d1ca-istio-token\") pod \"e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp\" (UID: \"9543233d-9115-4459-87b7-dea47aa8d1ca\") " pod="openshift-ingress/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp" Oct 30 20:08:56.085076 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:56.085022 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9543233d-9115-4459-87b7-dea47aa8d1ca-istio-data\") pod \"e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp\" (UID: \"9543233d-9115-4459-87b7-dea47aa8d1ca\") " pod="openshift-ingress/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp" Oct 30 20:08:56.085076 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:56.085053 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5cq9\" (UniqueName: \"kubernetes.io/projected/9543233d-9115-4459-87b7-dea47aa8d1ca-kube-api-access-q5cq9\") pod \"e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp\" (UID: \"9543233d-9115-4459-87b7-dea47aa8d1ca\") " pod="openshift-ingress/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp" Oct 30 20:08:56.085240 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:56.085149 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9543233d-9115-4459-87b7-dea47aa8d1ca-workload-certs\") pod \"e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp\" (UID: \"9543233d-9115-4459-87b7-dea47aa8d1ca\") " pod="openshift-ingress/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp" Oct 30 20:08:56.085240 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:56.085197 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9543233d-9115-4459-87b7-dea47aa8d1ca-workload-socket\") pod \"e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp\" (UID: \"9543233d-9115-4459-87b7-dea47aa8d1ca\") " pod="openshift-ingress/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp" Oct 30 20:08:56.085389 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:56.085278 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9543233d-9115-4459-87b7-dea47aa8d1ca-credential-socket\") pod \"e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp\" (UID: \"9543233d-9115-4459-87b7-dea47aa8d1ca\") " pod="openshift-ingress/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp" Oct 30 20:08:56.085389 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:56.085329 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9543233d-9115-4459-87b7-dea47aa8d1ca-istio-envoy\") pod \"e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp\" (UID: \"9543233d-9115-4459-87b7-dea47aa8d1ca\") " pod="openshift-ingress/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp" Oct 30 20:08:56.118936 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:56.118898 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp"] Oct 30 20:08:56.201031 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:56.200989 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9543233d-9115-4459-87b7-dea47aa8d1ca-istio-token\") pod \"e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp\" (UID: \"9543233d-9115-4459-87b7-dea47aa8d1ca\") " pod="openshift-ingress/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp" Oct 30 20:08:56.201228 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:56.201110 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9543233d-9115-4459-87b7-dea47aa8d1ca-istio-data\") pod \"e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp\" (UID: \"9543233d-9115-4459-87b7-dea47aa8d1ca\") " pod="openshift-ingress/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp" Oct 30 20:08:56.201228 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:56.201162 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5cq9\" (UniqueName: \"kubernetes.io/projected/9543233d-9115-4459-87b7-dea47aa8d1ca-kube-api-access-q5cq9\") pod \"e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp\" (UID: \"9543233d-9115-4459-87b7-dea47aa8d1ca\") " pod="openshift-ingress/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp" Oct 30 20:08:56.201228 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:56.201215 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9543233d-9115-4459-87b7-dea47aa8d1ca-workload-certs\") pod \"e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp\" (UID: \"9543233d-9115-4459-87b7-dea47aa8d1ca\") " pod="openshift-ingress/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp" Oct 30 20:08:56.201415 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:56.201243 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9543233d-9115-4459-87b7-dea47aa8d1ca-workload-socket\") pod \"e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp\" (UID: \"9543233d-9115-4459-87b7-dea47aa8d1ca\") " pod="openshift-ingress/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp" Oct 30 20:08:56.201415 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:56.201285 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9543233d-9115-4459-87b7-dea47aa8d1ca-credential-socket\") pod \"e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp\" (UID: \"9543233d-9115-4459-87b7-dea47aa8d1ca\") " pod="openshift-ingress/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp" Oct 30 20:08:56.201415 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:56.201338 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9543233d-9115-4459-87b7-dea47aa8d1ca-istio-envoy\") pod \"e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp\" (UID: \"9543233d-9115-4459-87b7-dea47aa8d1ca\") " pod="openshift-ingress/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp" Oct 30 20:08:56.201628 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:56.201467 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9543233d-9115-4459-87b7-dea47aa8d1ca-istio-podinfo\") pod \"e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp\" (UID: \"9543233d-9115-4459-87b7-dea47aa8d1ca\") " pod="openshift-ingress/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp" Oct 30 20:08:56.201628 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:56.201492 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9543233d-9115-4459-87b7-dea47aa8d1ca-istiod-ca-cert\") pod \"e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp\" (UID: \"9543233d-9115-4459-87b7-dea47aa8d1ca\") " pod="openshift-ingress/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp" Oct 30 20:08:56.201848 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:56.201817 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9543233d-9115-4459-87b7-dea47aa8d1ca-istio-data\") pod \"e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp\" (UID: \"9543233d-9115-4459-87b7-dea47aa8d1ca\") " pod="openshift-ingress/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp" Oct 30 20:08:56.202087 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:56.202043 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9543233d-9115-4459-87b7-dea47aa8d1ca-credential-socket\") pod \"e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp\" (UID: \"9543233d-9115-4459-87b7-dea47aa8d1ca\") " pod="openshift-ingress/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp" Oct 30 20:08:56.202366 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:56.202331 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9543233d-9115-4459-87b7-dea47aa8d1ca-workload-socket\") pod \"e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp\" (UID: \"9543233d-9115-4459-87b7-dea47aa8d1ca\") " pod="openshift-ingress/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp" Oct 30 20:08:56.203081 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:56.203058 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9543233d-9115-4459-87b7-dea47aa8d1ca-istiod-ca-cert\") pod \"e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp\" (UID: \"9543233d-9115-4459-87b7-dea47aa8d1ca\") " pod="openshift-ingress/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp" Oct 30 20:08:56.203310 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:56.203288 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9543233d-9115-4459-87b7-dea47aa8d1ca-workload-certs\") pod \"e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp\" (UID: \"9543233d-9115-4459-87b7-dea47aa8d1ca\") " pod="openshift-ingress/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp" Oct 30 20:08:56.207429 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:56.207042 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9543233d-9115-4459-87b7-dea47aa8d1ca-istio-podinfo\") pod \"e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp\" (UID: \"9543233d-9115-4459-87b7-dea47aa8d1ca\") " pod="openshift-ingress/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp" Oct 30 20:08:56.207429 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:56.207258 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9543233d-9115-4459-87b7-dea47aa8d1ca-istio-envoy\") pod \"e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp\" (UID: \"9543233d-9115-4459-87b7-dea47aa8d1ca\") " pod="openshift-ingress/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp" Oct 30 20:08:56.266734 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:56.266711 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9543233d-9115-4459-87b7-dea47aa8d1ca-istio-token\") pod \"e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp\" (UID: \"9543233d-9115-4459-87b7-dea47aa8d1ca\") " pod="openshift-ingress/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp" Oct 30 20:08:56.268852 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:56.268823 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5cq9\" (UniqueName: \"kubernetes.io/projected/9543233d-9115-4459-87b7-dea47aa8d1ca-kube-api-access-q5cq9\") pod \"e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp\" (UID: \"9543233d-9115-4459-87b7-dea47aa8d1ca\") " pod="openshift-ingress/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp" Oct 30 20:08:56.325859 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:56.325824 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp" Oct 30 20:08:56.769233 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:56.769046 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp"] Oct 30 20:08:56.772932 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:08:56.772901 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9543233d_9115_4459_87b7_dea47aa8d1ca.slice/crio-dd383518e772719114c20f50c5628fb8df910035bcd2b15f71afd209e0360706 WatchSource:0}: Error finding container dd383518e772719114c20f50c5628fb8df910035bcd2b15f71afd209e0360706: Status 404 returned error can't find the container with id dd383518e772719114c20f50c5628fb8df910035bcd2b15f71afd209e0360706 Oct 30 20:08:56.775954 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:56.775910 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:08:56.776049 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:56.775998 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:08:56.776049 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:56.776042 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:08:57.176981 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:57.176891 4292 scope.go:122] "RemoveContainer" containerID="9199cf36052feef81c8921548c3bf80f60d62b36c6f1e168967befd0bba1ebc7" Oct 30 20:08:57.448895 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:57.448812 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp" event={"ID":"9543233d-9115-4459-87b7-dea47aa8d1ca","Type":"ContainerStarted","Data":"8eaea5d77e405b49dc56c4751bd2ca3f9eb40b796c0e683291ea3a8780f150d5"} Oct 30 20:08:57.448895 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:57.448867 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp" event={"ID":"9543233d-9115-4459-87b7-dea47aa8d1ca","Type":"ContainerStarted","Data":"dd383518e772719114c20f50c5628fb8df910035bcd2b15f71afd209e0360706"} Oct 30 20:08:57.579901 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:57.579851 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp" podStartSLOduration=2.579835768 podStartE2EDuration="2.579835768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:08:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:08:57.577761576 +0000 UTC m=+2641.096077085" watchObservedRunningTime="2026-10-30 20:08:57.579835768 +0000 UTC m=+2641.098151225" Oct 30 20:08:58.326882 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:58.326851 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp" Oct 30 20:08:58.332142 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:58.332120 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp" Oct 30 20:08:58.455040 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:58.455011 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp" Oct 30 20:08:58.455929 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:08:58.455905 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp" Oct 30 20:09:13.220577 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:09:13.220486 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:09:43.220722 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:09:43.220683 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:09:57.211259 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:09:57.211221 4292 scope.go:122] "RemoveContainer" containerID="1940e2aeddd2e903771b23ff253ecd1b222c1d581a35454abd3513f2966ed81c" Oct 30 20:10:13.222319 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:10:13.222277 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:10:22.065664 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:22.065628 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eaa6d2-dlpwt"] Oct 30 20:10:22.069510 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:22.069492 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eaa6d2-dlpwt" Oct 30 20:10:22.126467 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:22.126435 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eaa6d2-dlpwt"] Oct 30 20:10:22.144730 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:22.144697 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8905c19-f981-4501-8fc1-0bfb9b5520ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-eaa6d2-dlpwt\" (UID: \"f8905c19-f981-4501-8fc1-0bfb9b5520ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eaa6d2-dlpwt" Oct 30 20:10:22.144892 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:22.144773 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8905c19-f981-4501-8fc1-0bfb9b5520ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-eaa6d2-dlpwt\" (UID: \"f8905c19-f981-4501-8fc1-0bfb9b5520ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eaa6d2-dlpwt" Oct 30 20:10:22.144892 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:22.144843 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4pd7\" (UniqueName: \"kubernetes.io/projected/f8905c19-f981-4501-8fc1-0bfb9b5520ef-kube-api-access-d4pd7\") pod \"maas-api-revoke-keys-e2e-ipp-a-eaa6d2-dlpwt\" (UID: \"f8905c19-f981-4501-8fc1-0bfb9b5520ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eaa6d2-dlpwt" Oct 30 20:10:22.173760 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:22.173725 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:10:22.173942 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:22.173791 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\"" Oct 30 20:10:22.246504 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:22.246475 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8905c19-f981-4501-8fc1-0bfb9b5520ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-eaa6d2-dlpwt\" (UID: \"f8905c19-f981-4501-8fc1-0bfb9b5520ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eaa6d2-dlpwt" Oct 30 20:10:22.246677 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:22.246509 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8905c19-f981-4501-8fc1-0bfb9b5520ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-eaa6d2-dlpwt\" (UID: \"f8905c19-f981-4501-8fc1-0bfb9b5520ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eaa6d2-dlpwt" Oct 30 20:10:22.246677 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:22.246570 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4pd7\" (UniqueName: \"kubernetes.io/projected/f8905c19-f981-4501-8fc1-0bfb9b5520ef-kube-api-access-d4pd7\") pod \"maas-api-revoke-keys-e2e-ipp-a-eaa6d2-dlpwt\" (UID: \"f8905c19-f981-4501-8fc1-0bfb9b5520ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eaa6d2-dlpwt" Oct 30 20:10:22.247124 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:22.247104 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8905c19-f981-4501-8fc1-0bfb9b5520ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-eaa6d2-dlpwt\" (UID: \"f8905c19-f981-4501-8fc1-0bfb9b5520ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eaa6d2-dlpwt" Oct 30 20:10:22.248941 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:22.248917 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8905c19-f981-4501-8fc1-0bfb9b5520ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-eaa6d2-dlpwt\" (UID: \"f8905c19-f981-4501-8fc1-0bfb9b5520ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eaa6d2-dlpwt" Oct 30 20:10:22.308901 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:22.308871 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4pd7\" (UniqueName: \"kubernetes.io/projected/f8905c19-f981-4501-8fc1-0bfb9b5520ef-kube-api-access-d4pd7\") pod \"maas-api-revoke-keys-e2e-ipp-a-eaa6d2-dlpwt\" (UID: \"f8905c19-f981-4501-8fc1-0bfb9b5520ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eaa6d2-dlpwt" Oct 30 20:10:22.382507 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:22.382421 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eaa6d2-dlpwt" Oct 30 20:10:22.758861 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:22.758836 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eaa6d2-dlpwt"] Oct 30 20:10:22.760455 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:10:22.760409 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8905c19_f981_4501_8fc1_0bfb9b5520ef.slice/crio-9d121145ffae22bfaf681079d75be43ce5e184acf2026cc23f660bafc2bdc6c5 WatchSource:0}: Error finding container 9d121145ffae22bfaf681079d75be43ce5e184acf2026cc23f660bafc2bdc6c5: Status 404 returned error can't find the container with id 9d121145ffae22bfaf681079d75be43ce5e184acf2026cc23f660bafc2bdc6c5 Oct 30 20:10:22.997109 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:22.997074 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eaa6d2-dlpwt" event={"ID":"f8905c19-f981-4501-8fc1-0bfb9b5520ef","Type":"ContainerStarted","Data":"9d121145ffae22bfaf681079d75be43ce5e184acf2026cc23f660bafc2bdc6c5"} Oct 30 20:10:24.005094 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:24.005001 4292 generic.go:356] "Generic (PLEG): container finished" podID="f8905c19-f981-4501-8fc1-0bfb9b5520ef" containerID="f205f99f62955a5e9b782fe3798cd80efc689feea87b0e956e19c94a7feb4987" exitCode=0 Oct 30 20:10:24.005540 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:24.005078 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eaa6d2-dlpwt" event={"ID":"f8905c19-f981-4501-8fc1-0bfb9b5520ef","Type":"ContainerDied","Data":"f205f99f62955a5e9b782fe3798cd80efc689feea87b0e956e19c94a7feb4987"} Oct 30 20:10:25.152672 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:25.152646 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eaa6d2-dlpwt" Oct 30 20:10:25.277260 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:25.277210 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8905c19-f981-4501-8fc1-0bfb9b5520ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8905c19-f981-4501-8fc1-0bfb9b5520ef-maas-api-openshift-ca\") pod \"f8905c19-f981-4501-8fc1-0bfb9b5520ef\" (UID: \"f8905c19-f981-4501-8fc1-0bfb9b5520ef\") " Oct 30 20:10:25.277260 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:25.277255 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8905c19-f981-4501-8fc1-0bfb9b5520ef-kube-api-access-d4pd7\" (UniqueName: \"kubernetes.io/projected/f8905c19-f981-4501-8fc1-0bfb9b5520ef-kube-api-access-d4pd7\") pod \"f8905c19-f981-4501-8fc1-0bfb9b5520ef\" (UID: \"f8905c19-f981-4501-8fc1-0bfb9b5520ef\") " Oct 30 20:10:25.277627 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:25.277296 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8905c19-f981-4501-8fc1-0bfb9b5520ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8905c19-f981-4501-8fc1-0bfb9b5520ef-maas-api-xks-ca\") pod \"f8905c19-f981-4501-8fc1-0bfb9b5520ef\" (UID: \"f8905c19-f981-4501-8fc1-0bfb9b5520ef\") " Oct 30 20:10:25.277977 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:25.277944 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8905c19-f981-4501-8fc1-0bfb9b5520ef-maas-api-openshift-ca" pod "f8905c19-f981-4501-8fc1-0bfb9b5520ef" (UID: "f8905c19-f981-4501-8fc1-0bfb9b5520ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:10:25.279568 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:25.279538 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8905c19-f981-4501-8fc1-0bfb9b5520ef-maas-api-xks-ca" pod "f8905c19-f981-4501-8fc1-0bfb9b5520ef" (UID: "f8905c19-f981-4501-8fc1-0bfb9b5520ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:10:25.279667 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:25.279622 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8905c19-f981-4501-8fc1-0bfb9b5520ef-kube-api-access-d4pd7" pod "f8905c19-f981-4501-8fc1-0bfb9b5520ef" (UID: "f8905c19-f981-4501-8fc1-0bfb9b5520ef"). InnerVolumeSpecName "kube-api-access-d4pd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:10:25.378934 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:25.378894 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8905c19-f981-4501-8fc1-0bfb9b5520ef-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:25.378934 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:25.378927 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4pd7\" (UniqueName: \"kubernetes.io/projected/f8905c19-f981-4501-8fc1-0bfb9b5520ef-kube-api-access-d4pd7\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:25.379203 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:25.378942 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8905c19-f981-4501-8fc1-0bfb9b5520ef-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:26.017139 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:26.017107 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eaa6d2-dlpwt" event={"ID":"f8905c19-f981-4501-8fc1-0bfb9b5520ef","Type":"ContainerDied","Data":"9d121145ffae22bfaf681079d75be43ce5e184acf2026cc23f660bafc2bdc6c5"} Oct 30 20:10:26.017139 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:26.017132 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eaa6d2-dlpwt" Oct 30 20:10:26.017139 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:26.017155 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d121145ffae22bfaf681079d75be43ce5e184acf2026cc23f660bafc2bdc6c5" Oct 30 20:10:28.684299 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:28.684263 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml"] Oct 30 20:10:28.684838 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:28.684524 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml" podUID="69d11c12-c4cb-452e-90cc-4cbe03333951" containerName="istio-proxy" containerID="cri-o://90bddd7abe3fb567b216e0ce5a0a6e4226b0e4e16c1fcf8e001744bd1ed7b18b" gracePeriod=30 Oct 30 20:10:28.713404 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:28.713370 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x"] Oct 30 20:10:28.713620 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:28.713594 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x" podUID="931c961c-cdf7-442e-8a6f-ff766e813cad" containerName="istio-proxy" containerID="cri-o://8a7ea8f7f609e26bbb5bc4554256a089c7cc68afae431b9ffff3129e79456c20" gracePeriod=30 Oct 30 20:10:30.514521 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:30.514478 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8nrtp"] Oct 30 20:10:30.514947 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:30.514775 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8nrtp" podUID="b5978bc5-3ff8-4f4a-b85f-58b737179fee" containerName="istio-proxy" containerID="cri-o://7ae7bbea353c7a21ae7b84d11a3cd62c3b88e4c4a1907e08b9babfb70f3b335f" gracePeriod=30 Oct 30 20:10:33.968175 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:33.968110 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml" Oct 30 20:10:34.009832 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.009807 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x" Oct 30 20:10:34.073744 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.073702 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69d11c12-c4cb-452e-90cc-4cbe03333951-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69d11c12-c4cb-452e-90cc-4cbe03333951-workload-certs\") pod \"69d11c12-c4cb-452e-90cc-4cbe03333951\" (UID: \"69d11c12-c4cb-452e-90cc-4cbe03333951\") " Oct 30 20:10:34.073960 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.073766 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69d11c12-c4cb-452e-90cc-4cbe03333951-istio-token\" (UniqueName: \"kubernetes.io/projected/69d11c12-c4cb-452e-90cc-4cbe03333951-istio-token\") pod \"69d11c12-c4cb-452e-90cc-4cbe03333951\" (UID: \"69d11c12-c4cb-452e-90cc-4cbe03333951\") " Oct 30 20:10:34.073960 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.073802 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69d11c12-c4cb-452e-90cc-4cbe03333951-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69d11c12-c4cb-452e-90cc-4cbe03333951-istiod-ca-cert\") pod \"69d11c12-c4cb-452e-90cc-4cbe03333951\" (UID: \"69d11c12-c4cb-452e-90cc-4cbe03333951\") " Oct 30 20:10:34.073960 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.073837 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/931c961c-cdf7-442e-8a6f-ff766e813cad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/931c961c-cdf7-442e-8a6f-ff766e813cad-istio-data\") pod \"931c961c-cdf7-442e-8a6f-ff766e813cad\" (UID: \"931c961c-cdf7-442e-8a6f-ff766e813cad\") " Oct 30 20:10:34.073960 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.073901 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/931c961c-cdf7-442e-8a6f-ff766e813cad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/931c961c-cdf7-442e-8a6f-ff766e813cad-workload-certs\") pod \"931c961c-cdf7-442e-8a6f-ff766e813cad\" (UID: \"931c961c-cdf7-442e-8a6f-ff766e813cad\") " Oct 30 20:10:34.073960 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.073938 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69d11c12-c4cb-452e-90cc-4cbe03333951-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69d11c12-c4cb-452e-90cc-4cbe03333951-istio-data\") pod \"69d11c12-c4cb-452e-90cc-4cbe03333951\" (UID: \"69d11c12-c4cb-452e-90cc-4cbe03333951\") " Oct 30 20:10:34.074258 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.073974 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/931c961c-cdf7-442e-8a6f-ff766e813cad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/931c961c-cdf7-442e-8a6f-ff766e813cad-istio-envoy\") pod \"931c961c-cdf7-442e-8a6f-ff766e813cad\" (UID: \"931c961c-cdf7-442e-8a6f-ff766e813cad\") " Oct 30 20:10:34.074258 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.074018 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69d11c12-c4cb-452e-90cc-4cbe03333951-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69d11c12-c4cb-452e-90cc-4cbe03333951-istio-podinfo\") pod \"69d11c12-c4cb-452e-90cc-4cbe03333951\" (UID: \"69d11c12-c4cb-452e-90cc-4cbe03333951\") " Oct 30 20:10:34.074258 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.074045 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69d11c12-c4cb-452e-90cc-4cbe03333951-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69d11c12-c4cb-452e-90cc-4cbe03333951-workload-socket\") pod \"69d11c12-c4cb-452e-90cc-4cbe03333951\" (UID: \"69d11c12-c4cb-452e-90cc-4cbe03333951\") " Oct 30 20:10:34.074258 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.074096 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69d11c12-c4cb-452e-90cc-4cbe03333951-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69d11c12-c4cb-452e-90cc-4cbe03333951-credential-socket\") pod \"69d11c12-c4cb-452e-90cc-4cbe03333951\" (UID: \"69d11c12-c4cb-452e-90cc-4cbe03333951\") " Oct 30 20:10:34.074258 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.074140 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/931c961c-cdf7-442e-8a6f-ff766e813cad-istio-token\" (UniqueName: \"kubernetes.io/projected/931c961c-cdf7-442e-8a6f-ff766e813cad-istio-token\") pod \"931c961c-cdf7-442e-8a6f-ff766e813cad\" (UID: \"931c961c-cdf7-442e-8a6f-ff766e813cad\") " Oct 30 20:10:34.074258 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.074190 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69d11c12-c4cb-452e-90cc-4cbe03333951-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69d11c12-c4cb-452e-90cc-4cbe03333951-istio-envoy\") pod \"69d11c12-c4cb-452e-90cc-4cbe03333951\" (UID: \"69d11c12-c4cb-452e-90cc-4cbe03333951\") " Oct 30 20:10:34.074258 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.074223 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/931c961c-cdf7-442e-8a6f-ff766e813cad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/931c961c-cdf7-442e-8a6f-ff766e813cad-istio-podinfo\") pod \"931c961c-cdf7-442e-8a6f-ff766e813cad\" (UID: \"931c961c-cdf7-442e-8a6f-ff766e813cad\") " Oct 30 20:10:34.074652 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.074271 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/931c961c-cdf7-442e-8a6f-ff766e813cad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/931c961c-cdf7-442e-8a6f-ff766e813cad-workload-socket\") pod \"931c961c-cdf7-442e-8a6f-ff766e813cad\" (UID: \"931c961c-cdf7-442e-8a6f-ff766e813cad\") " Oct 30 20:10:34.074652 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.074310 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/931c961c-cdf7-442e-8a6f-ff766e813cad-kube-api-access-hh6kg\" (UniqueName: \"kubernetes.io/projected/931c961c-cdf7-442e-8a6f-ff766e813cad-kube-api-access-hh6kg\") pod \"931c961c-cdf7-442e-8a6f-ff766e813cad\" (UID: \"931c961c-cdf7-442e-8a6f-ff766e813cad\") " Oct 30 20:10:34.074652 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.074379 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/931c961c-cdf7-442e-8a6f-ff766e813cad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/931c961c-cdf7-442e-8a6f-ff766e813cad-istiod-ca-cert\") pod \"931c961c-cdf7-442e-8a6f-ff766e813cad\" (UID: \"931c961c-cdf7-442e-8a6f-ff766e813cad\") " Oct 30 20:10:34.074652 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.074435 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69d11c12-c4cb-452e-90cc-4cbe03333951-kube-api-access-x2nz5\" (UniqueName: \"kubernetes.io/projected/69d11c12-c4cb-452e-90cc-4cbe03333951-kube-api-access-x2nz5\") pod \"69d11c12-c4cb-452e-90cc-4cbe03333951\" (UID: \"69d11c12-c4cb-452e-90cc-4cbe03333951\") " Oct 30 20:10:34.074652 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.074496 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/931c961c-cdf7-442e-8a6f-ff766e813cad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/931c961c-cdf7-442e-8a6f-ff766e813cad-credential-socket\") pod \"931c961c-cdf7-442e-8a6f-ff766e813cad\" (UID: \"931c961c-cdf7-442e-8a6f-ff766e813cad\") " Oct 30 20:10:34.075221 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.075193 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/931c961c-cdf7-442e-8a6f-ff766e813cad-credential-socket" pod "931c961c-cdf7-442e-8a6f-ff766e813cad" (UID: "931c961c-cdf7-442e-8a6f-ff766e813cad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:34.075488 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.075282 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/931c961c-cdf7-442e-8a6f-ff766e813cad-istiod-ca-cert" pod "931c961c-cdf7-442e-8a6f-ff766e813cad" (UID: "931c961c-cdf7-442e-8a6f-ff766e813cad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:10:34.076143 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.076119 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69d11c12-c4cb-452e-90cc-4cbe03333951-workload-certs" pod "69d11c12-c4cb-452e-90cc-4cbe03333951" (UID: "69d11c12-c4cb-452e-90cc-4cbe03333951"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:34.076493 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.076473 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/931c961c-cdf7-442e-8a6f-ff766e813cad-workload-certs" pod "931c961c-cdf7-442e-8a6f-ff766e813cad" (UID: "931c961c-cdf7-442e-8a6f-ff766e813cad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:34.076871 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.076844 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69d11c12-c4cb-452e-90cc-4cbe03333951-istiod-ca-cert" pod "69d11c12-c4cb-452e-90cc-4cbe03333951" (UID: "69d11c12-c4cb-452e-90cc-4cbe03333951"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:10:34.077663 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.077451 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/931c961c-cdf7-442e-8a6f-ff766e813cad-istio-data" pod "931c961c-cdf7-442e-8a6f-ff766e813cad" (UID: "931c961c-cdf7-442e-8a6f-ff766e813cad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:34.078130 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.078108 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/931c961c-cdf7-442e-8a6f-ff766e813cad-workload-socket" pod "931c961c-cdf7-442e-8a6f-ff766e813cad" (UID: "931c961c-cdf7-442e-8a6f-ff766e813cad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:34.078511 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.078491 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69d11c12-c4cb-452e-90cc-4cbe03333951-istio-data" pod "69d11c12-c4cb-452e-90cc-4cbe03333951" (UID: "69d11c12-c4cb-452e-90cc-4cbe03333951"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:34.078788 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.078766 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69d11c12-c4cb-452e-90cc-4cbe03333951-workload-socket" pod "69d11c12-c4cb-452e-90cc-4cbe03333951" (UID: "69d11c12-c4cb-452e-90cc-4cbe03333951"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:34.078949 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.078928 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69d11c12-c4cb-452e-90cc-4cbe03333951-credential-socket" pod "69d11c12-c4cb-452e-90cc-4cbe03333951" (UID: "69d11c12-c4cb-452e-90cc-4cbe03333951"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:34.080822 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.080783 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/931c961c-cdf7-442e-8a6f-ff766e813cad-istio-podinfo" pod "931c961c-cdf7-442e-8a6f-ff766e813cad" (UID: "931c961c-cdf7-442e-8a6f-ff766e813cad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:10:34.082939 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.082430 4292 generic.go:356] "Generic (PLEG): container finished" podID="931c961c-cdf7-442e-8a6f-ff766e813cad" containerID="8a7ea8f7f609e26bbb5bc4554256a089c7cc68afae431b9ffff3129e79456c20" exitCode=0 Oct 30 20:10:34.082939 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.082643 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x" Oct 30 20:10:34.082939 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.082787 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/931c961c-cdf7-442e-8a6f-ff766e813cad-istio-envoy" pod "931c961c-cdf7-442e-8a6f-ff766e813cad" (UID: "931c961c-cdf7-442e-8a6f-ff766e813cad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:34.082939 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.082800 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/931c961c-cdf7-442e-8a6f-ff766e813cad-kube-api-access-hh6kg" pod "931c961c-cdf7-442e-8a6f-ff766e813cad" (UID: "931c961c-cdf7-442e-8a6f-ff766e813cad"). InnerVolumeSpecName "kube-api-access-hh6kg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:10:34.082939 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.082796 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x" event={"ID":"931c961c-cdf7-442e-8a6f-ff766e813cad","Type":"ContainerDied","Data":"8a7ea8f7f609e26bbb5bc4554256a089c7cc68afae431b9ffff3129e79456c20"} Oct 30 20:10:34.082939 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.082866 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69d11c12-c4cb-452e-90cc-4cbe03333951-istio-token" pod "69d11c12-c4cb-452e-90cc-4cbe03333951" (UID: "69d11c12-c4cb-452e-90cc-4cbe03333951"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:10:34.082939 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.082868 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69d11c12-c4cb-452e-90cc-4cbe03333951-istio-envoy" pod "69d11c12-c4cb-452e-90cc-4cbe03333951" (UID: "69d11c12-c4cb-452e-90cc-4cbe03333951"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:34.082939 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.082873 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x" event={"ID":"931c961c-cdf7-442e-8a6f-ff766e813cad","Type":"ContainerDied","Data":"79e89fbd3ac0387c8bd20f2af185da9b88697b1b01e0622e07aef562acad45fe"} Oct 30 20:10:34.082939 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.082892 4292 scope.go:122] "RemoveContainer" containerID="8a7ea8f7f609e26bbb5bc4554256a089c7cc68afae431b9ffff3129e79456c20" Oct 30 20:10:34.083655 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.083263 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69d11c12-c4cb-452e-90cc-4cbe03333951-istio-podinfo" pod "69d11c12-c4cb-452e-90cc-4cbe03333951" (UID: "69d11c12-c4cb-452e-90cc-4cbe03333951"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:10:34.083752 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.083730 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69d11c12-c4cb-452e-90cc-4cbe03333951-kube-api-access-x2nz5" pod "69d11c12-c4cb-452e-90cc-4cbe03333951" (UID: "69d11c12-c4cb-452e-90cc-4cbe03333951"). InnerVolumeSpecName "kube-api-access-x2nz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:10:34.084615 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.084596 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/931c961c-cdf7-442e-8a6f-ff766e813cad-istio-token" pod "931c961c-cdf7-442e-8a6f-ff766e813cad" (UID: "931c961c-cdf7-442e-8a6f-ff766e813cad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:10:34.084803 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.084780 4292 generic.go:356] "Generic (PLEG): container finished" podID="69d11c12-c4cb-452e-90cc-4cbe03333951" containerID="90bddd7abe3fb567b216e0ce5a0a6e4226b0e4e16c1fcf8e001744bd1ed7b18b" exitCode=0 Oct 30 20:10:34.084907 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.084829 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml" Oct 30 20:10:34.084907 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.084846 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml" event={"ID":"69d11c12-c4cb-452e-90cc-4cbe03333951","Type":"ContainerDied","Data":"90bddd7abe3fb567b216e0ce5a0a6e4226b0e4e16c1fcf8e001744bd1ed7b18b"} Oct 30 20:10:34.084982 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.084905 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml" event={"ID":"69d11c12-c4cb-452e-90cc-4cbe03333951","Type":"ContainerDied","Data":"892028e27c2c1381e5591f3137d7fa9e42246b8c1765844d2125be6c46c37521"} Oct 30 20:10:34.093089 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.093069 4292 scope.go:122] "RemoveContainer" containerID="8a7ea8f7f609e26bbb5bc4554256a089c7cc68afae431b9ffff3129e79456c20" Oct 30 20:10:34.093431 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:10:34.093411 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a7ea8f7f609e26bbb5bc4554256a089c7cc68afae431b9ffff3129e79456c20\": container with ID starting with 8a7ea8f7f609e26bbb5bc4554256a089c7cc68afae431b9ffff3129e79456c20 not found: ID does not exist" containerID="8a7ea8f7f609e26bbb5bc4554256a089c7cc68afae431b9ffff3129e79456c20" Oct 30 20:10:34.093533 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.093442 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a7ea8f7f609e26bbb5bc4554256a089c7cc68afae431b9ffff3129e79456c20"} err="failed to get container status \"8a7ea8f7f609e26bbb5bc4554256a089c7cc68afae431b9ffff3129e79456c20\": rpc error: code = NotFound desc = could not find container \"8a7ea8f7f609e26bbb5bc4554256a089c7cc68afae431b9ffff3129e79456c20\": container with ID starting with 8a7ea8f7f609e26bbb5bc4554256a089c7cc68afae431b9ffff3129e79456c20 not found: ID does not exist" Oct 30 20:10:34.093533 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.093468 4292 scope.go:122] "RemoveContainer" containerID="90bddd7abe3fb567b216e0ce5a0a6e4226b0e4e16c1fcf8e001744bd1ed7b18b" Oct 30 20:10:34.101903 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.101878 4292 scope.go:122] "RemoveContainer" containerID="90bddd7abe3fb567b216e0ce5a0a6e4226b0e4e16c1fcf8e001744bd1ed7b18b" Oct 30 20:10:34.102264 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:10:34.102246 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90bddd7abe3fb567b216e0ce5a0a6e4226b0e4e16c1fcf8e001744bd1ed7b18b\": container with ID starting with 90bddd7abe3fb567b216e0ce5a0a6e4226b0e4e16c1fcf8e001744bd1ed7b18b not found: ID does not exist" containerID="90bddd7abe3fb567b216e0ce5a0a6e4226b0e4e16c1fcf8e001744bd1ed7b18b" Oct 30 20:10:34.102315 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.102273 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90bddd7abe3fb567b216e0ce5a0a6e4226b0e4e16c1fcf8e001744bd1ed7b18b"} err="failed to get container status \"90bddd7abe3fb567b216e0ce5a0a6e4226b0e4e16c1fcf8e001744bd1ed7b18b\": rpc error: code = NotFound desc = could not find container \"90bddd7abe3fb567b216e0ce5a0a6e4226b0e4e16c1fcf8e001744bd1ed7b18b\": container with ID starting with 90bddd7abe3fb567b216e0ce5a0a6e4226b0e4e16c1fcf8e001744bd1ed7b18b not found: ID does not exist" Oct 30 20:10:34.176178 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.176143 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69d11c12-c4cb-452e-90cc-4cbe03333951-istio-data\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:34.176178 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.176179 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/931c961c-cdf7-442e-8a6f-ff766e813cad-istio-envoy\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:34.176495 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.176195 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69d11c12-c4cb-452e-90cc-4cbe03333951-istio-podinfo\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:34.176495 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.176208 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69d11c12-c4cb-452e-90cc-4cbe03333951-workload-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:34.176495 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.176221 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69d11c12-c4cb-452e-90cc-4cbe03333951-credential-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:34.176495 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.176233 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/931c961c-cdf7-442e-8a6f-ff766e813cad-istio-token\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:34.176495 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.176246 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69d11c12-c4cb-452e-90cc-4cbe03333951-istio-envoy\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:34.176495 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.176258 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/931c961c-cdf7-442e-8a6f-ff766e813cad-istio-podinfo\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:34.176495 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.176267 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/931c961c-cdf7-442e-8a6f-ff766e813cad-workload-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:34.176495 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.176276 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hh6kg\" (UniqueName: \"kubernetes.io/projected/931c961c-cdf7-442e-8a6f-ff766e813cad-kube-api-access-hh6kg\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:34.176495 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.176288 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/931c961c-cdf7-442e-8a6f-ff766e813cad-istiod-ca-cert\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:34.176495 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.176296 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2nz5\" (UniqueName: \"kubernetes.io/projected/69d11c12-c4cb-452e-90cc-4cbe03333951-kube-api-access-x2nz5\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:34.176495 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.176305 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/931c961c-cdf7-442e-8a6f-ff766e813cad-credential-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:34.176495 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.176313 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69d11c12-c4cb-452e-90cc-4cbe03333951-workload-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:34.176495 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.176323 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69d11c12-c4cb-452e-90cc-4cbe03333951-istio-token\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:34.176495 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.176331 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69d11c12-c4cb-452e-90cc-4cbe03333951-istiod-ca-cert\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:34.176495 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.176339 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/931c961c-cdf7-442e-8a6f-ff766e813cad-istio-data\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:34.176495 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.176375 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/931c961c-cdf7-442e-8a6f-ff766e813cad-workload-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:34.260232 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.260088 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml"] Oct 30 20:10:34.264815 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.264739 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-q76ml"] Oct 30 20:10:34.526778 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.526744 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x"] Oct 30 20:10:34.529186 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.529159 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-m6b9x"] Oct 30 20:10:34.584825 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.584786 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69d11c12-c4cb-452e-90cc-4cbe03333951" path="/var/lib/kubelet/pods/69d11c12-c4cb-452e-90cc-4cbe03333951/volumes" Oct 30 20:10:34.585376 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:34.585338 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="931c961c-cdf7-442e-8a6f-ff766e813cad" path="/var/lib/kubelet/pods/931c961c-cdf7-442e-8a6f-ff766e813cad/volumes" Oct 30 20:10:35.824065 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:35.824034 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8nrtp" Oct 30 20:10:35.909793 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:35.909756 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5978bc5-3ff8-4f4a-b85f-58b737179fee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5978bc5-3ff8-4f4a-b85f-58b737179fee-workload-socket\") pod \"b5978bc5-3ff8-4f4a-b85f-58b737179fee\" (UID: \"b5978bc5-3ff8-4f4a-b85f-58b737179fee\") " Oct 30 20:10:35.909991 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:35.909816 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5978bc5-3ff8-4f4a-b85f-58b737179fee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5978bc5-3ff8-4f4a-b85f-58b737179fee-istio-envoy\") pod \"b5978bc5-3ff8-4f4a-b85f-58b737179fee\" (UID: \"b5978bc5-3ff8-4f4a-b85f-58b737179fee\") " Oct 30 20:10:35.909991 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:35.909854 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5978bc5-3ff8-4f4a-b85f-58b737179fee-istio-token\" (UniqueName: \"kubernetes.io/projected/b5978bc5-3ff8-4f4a-b85f-58b737179fee-istio-token\") pod \"b5978bc5-3ff8-4f4a-b85f-58b737179fee\" (UID: \"b5978bc5-3ff8-4f4a-b85f-58b737179fee\") " Oct 30 20:10:35.909991 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:35.909888 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5978bc5-3ff8-4f4a-b85f-58b737179fee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5978bc5-3ff8-4f4a-b85f-58b737179fee-credential-socket\") pod \"b5978bc5-3ff8-4f4a-b85f-58b737179fee\" (UID: \"b5978bc5-3ff8-4f4a-b85f-58b737179fee\") " Oct 30 20:10:35.909991 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:35.909941 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5978bc5-3ff8-4f4a-b85f-58b737179fee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5978bc5-3ff8-4f4a-b85f-58b737179fee-istio-data\") pod \"b5978bc5-3ff8-4f4a-b85f-58b737179fee\" (UID: \"b5978bc5-3ff8-4f4a-b85f-58b737179fee\") " Oct 30 20:10:35.910220 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:35.910069 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5978bc5-3ff8-4f4a-b85f-58b737179fee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5978bc5-3ff8-4f4a-b85f-58b737179fee-istiod-ca-cert\") pod \"b5978bc5-3ff8-4f4a-b85f-58b737179fee\" (UID: \"b5978bc5-3ff8-4f4a-b85f-58b737179fee\") " Oct 30 20:10:35.910220 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:35.910107 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5978bc5-3ff8-4f4a-b85f-58b737179fee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5978bc5-3ff8-4f4a-b85f-58b737179fee-workload-certs\") pod \"b5978bc5-3ff8-4f4a-b85f-58b737179fee\" (UID: \"b5978bc5-3ff8-4f4a-b85f-58b737179fee\") " Oct 30 20:10:35.910220 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:35.910169 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b5978bc5-3ff8-4f4a-b85f-58b737179fee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5978bc5-3ff8-4f4a-b85f-58b737179fee-istio-podinfo\") pod \"b5978bc5-3ff8-4f4a-b85f-58b737179fee\" (UID: \"b5978bc5-3ff8-4f4a-b85f-58b737179fee\") " Oct 30 20:10:35.910406 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:35.910225 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5978bc5-3ff8-4f4a-b85f-58b737179fee-kube-api-access-6tng4\" (UniqueName: \"kubernetes.io/projected/b5978bc5-3ff8-4f4a-b85f-58b737179fee-kube-api-access-6tng4\") pod \"b5978bc5-3ff8-4f4a-b85f-58b737179fee\" (UID: \"b5978bc5-3ff8-4f4a-b85f-58b737179fee\") " Oct 30 20:10:35.910516 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:35.910064 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5978bc5-3ff8-4f4a-b85f-58b737179fee-workload-socket" pod "b5978bc5-3ff8-4f4a-b85f-58b737179fee" (UID: "b5978bc5-3ff8-4f4a-b85f-58b737179fee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:35.910516 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:35.910202 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5978bc5-3ff8-4f4a-b85f-58b737179fee-credential-socket" pod "b5978bc5-3ff8-4f4a-b85f-58b737179fee" (UID: "b5978bc5-3ff8-4f4a-b85f-58b737179fee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:35.910516 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:35.910465 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5978bc5-3ff8-4f4a-b85f-58b737179fee-istiod-ca-cert" pod "b5978bc5-3ff8-4f4a-b85f-58b737179fee" (UID: "b5978bc5-3ff8-4f4a-b85f-58b737179fee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:10:35.910708 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:35.910507 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5978bc5-3ff8-4f4a-b85f-58b737179fee-istio-data" pod "b5978bc5-3ff8-4f4a-b85f-58b737179fee" (UID: "b5978bc5-3ff8-4f4a-b85f-58b737179fee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:35.910772 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:35.910738 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5978bc5-3ff8-4f4a-b85f-58b737179fee-istiod-ca-cert\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:35.910772 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:35.910759 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5978bc5-3ff8-4f4a-b85f-58b737179fee-workload-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:35.910872 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:35.910774 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5978bc5-3ff8-4f4a-b85f-58b737179fee-credential-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:35.910872 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:35.910817 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5978bc5-3ff8-4f4a-b85f-58b737179fee-istio-data\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:35.910963 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:35.910860 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5978bc5-3ff8-4f4a-b85f-58b737179fee-workload-certs" pod "b5978bc5-3ff8-4f4a-b85f-58b737179fee" (UID: "b5978bc5-3ff8-4f4a-b85f-58b737179fee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:35.913089 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:35.913060 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5978bc5-3ff8-4f4a-b85f-58b737179fee-istio-podinfo" pod "b5978bc5-3ff8-4f4a-b85f-58b737179fee" (UID: "b5978bc5-3ff8-4f4a-b85f-58b737179fee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:10:35.913184 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:35.913088 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5978bc5-3ff8-4f4a-b85f-58b737179fee-istio-envoy" pod "b5978bc5-3ff8-4f4a-b85f-58b737179fee" (UID: "b5978bc5-3ff8-4f4a-b85f-58b737179fee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:35.913326 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:35.913297 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5978bc5-3ff8-4f4a-b85f-58b737179fee-kube-api-access-6tng4" pod "b5978bc5-3ff8-4f4a-b85f-58b737179fee" (UID: "b5978bc5-3ff8-4f4a-b85f-58b737179fee"). InnerVolumeSpecName "kube-api-access-6tng4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:10:35.914217 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:35.914197 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5978bc5-3ff8-4f4a-b85f-58b737179fee-istio-token" pod "b5978bc5-3ff8-4f4a-b85f-58b737179fee" (UID: "b5978bc5-3ff8-4f4a-b85f-58b737179fee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:10:36.012081 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:36.012045 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tng4\" (UniqueName: \"kubernetes.io/projected/b5978bc5-3ff8-4f4a-b85f-58b737179fee-kube-api-access-6tng4\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:36.012081 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:36.012076 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5978bc5-3ff8-4f4a-b85f-58b737179fee-istio-envoy\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:36.012081 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:36.012085 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5978bc5-3ff8-4f4a-b85f-58b737179fee-istio-token\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:36.012396 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:36.012093 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5978bc5-3ff8-4f4a-b85f-58b737179fee-workload-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:36.012396 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:36.012103 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5978bc5-3ff8-4f4a-b85f-58b737179fee-istio-podinfo\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:36.099854 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:36.099766 4292 generic.go:356] "Generic (PLEG): container finished" podID="b5978bc5-3ff8-4f4a-b85f-58b737179fee" containerID="7ae7bbea353c7a21ae7b84d11a3cd62c3b88e4c4a1907e08b9babfb70f3b335f" exitCode=0 Oct 30 20:10:36.099854 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:36.099820 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8nrtp" Oct 30 20:10:36.100073 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:36.099857 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8nrtp" event={"ID":"b5978bc5-3ff8-4f4a-b85f-58b737179fee","Type":"ContainerDied","Data":"7ae7bbea353c7a21ae7b84d11a3cd62c3b88e4c4a1907e08b9babfb70f3b335f"} Oct 30 20:10:36.100073 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:36.099953 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8nrtp" event={"ID":"b5978bc5-3ff8-4f4a-b85f-58b737179fee","Type":"ContainerDied","Data":"b27b411ed02fe38282c171739f425f4d9ec1d7fd15888ac44d5d51c4dd7cc25d"} Oct 30 20:10:36.100073 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:36.099986 4292 scope.go:122] "RemoveContainer" containerID="7ae7bbea353c7a21ae7b84d11a3cd62c3b88e4c4a1907e08b9babfb70f3b335f" Oct 30 20:10:36.116625 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:36.116608 4292 scope.go:122] "RemoveContainer" containerID="7ae7bbea353c7a21ae7b84d11a3cd62c3b88e4c4a1907e08b9babfb70f3b335f" Oct 30 20:10:36.116934 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:10:36.116908 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ae7bbea353c7a21ae7b84d11a3cd62c3b88e4c4a1907e08b9babfb70f3b335f\": container with ID starting with 7ae7bbea353c7a21ae7b84d11a3cd62c3b88e4c4a1907e08b9babfb70f3b335f not found: ID does not exist" containerID="7ae7bbea353c7a21ae7b84d11a3cd62c3b88e4c4a1907e08b9babfb70f3b335f" Oct 30 20:10:36.117005 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:36.116947 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ae7bbea353c7a21ae7b84d11a3cd62c3b88e4c4a1907e08b9babfb70f3b335f"} err="failed to get container status \"7ae7bbea353c7a21ae7b84d11a3cd62c3b88e4c4a1907e08b9babfb70f3b335f\": rpc error: code = NotFound desc = could not find container \"7ae7bbea353c7a21ae7b84d11a3cd62c3b88e4c4a1907e08b9babfb70f3b335f\": container with ID starting with 7ae7bbea353c7a21ae7b84d11a3cd62c3b88e4c4a1907e08b9babfb70f3b335f not found: ID does not exist" Oct 30 20:10:36.271765 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:36.271726 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8nrtp"] Oct 30 20:10:36.276375 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:36.275683 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-8nrtp"] Oct 30 20:10:36.586941 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:36.586908 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5978bc5-3ff8-4f4a-b85f-58b737179fee" path="/var/lib/kubelet/pods/b5978bc5-3ff8-4f4a-b85f-58b737179fee/volumes" Oct 30 20:10:43.149127 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:43.149095 4292 generic.go:356] "Generic (PLEG): container finished" podID="9543233d-9115-4459-87b7-dea47aa8d1ca" containerID="8eaea5d77e405b49dc56c4751bd2ca3f9eb40b796c0e683291ea3a8780f150d5" exitCode=0 Oct 30 20:10:43.149580 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:43.149134 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp" event={"ID":"9543233d-9115-4459-87b7-dea47aa8d1ca","Type":"ContainerDied","Data":"8eaea5d77e405b49dc56c4751bd2ca3f9eb40b796c0e683291ea3a8780f150d5"} Oct 30 20:10:43.149580 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:43.149517 4292 scope.go:122] "RemoveContainer" containerID="8eaea5d77e405b49dc56c4751bd2ca3f9eb40b796c0e683291ea3a8780f150d5" Oct 30 20:10:43.151012 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:43.150986 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:10:43.151071 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:43.151052 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:10:43.151105 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:43.151085 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:10:43.222495 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:10:43.222462 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:10:43.455057 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:43.454983 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp" Oct 30 20:10:44.175500 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:44.175458 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp" event={"ID":"9543233d-9115-4459-87b7-dea47aa8d1ca","Type":"ContainerStarted","Data":"7853d32bb00ffae07b1f5a18938660074982e78890b8f0d7e2c9745c84cef977"} Oct 30 20:10:44.326104 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:44.326070 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp" Oct 30 20:10:44.327499 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:44.327471 4292 patch_prober.go:29] interesting pod/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 30 20:10:44.327650 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:44.327529 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp" podUID="9543233d-9115-4459-87b7-dea47aa8d1ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 30 20:10:45.327088 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:45.327047 4292 patch_prober.go:29] interesting pod/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 30 20:10:45.327534 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:45.327109 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp" podUID="9543233d-9115-4459-87b7-dea47aa8d1ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 30 20:10:46.326776 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:46.326743 4292 patch_prober.go:29] interesting pod/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 30 20:10:46.326966 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:46.326798 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp" podUID="9543233d-9115-4459-87b7-dea47aa8d1ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 30 20:10:47.327077 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:47.327039 4292 patch_prober.go:29] interesting pod/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 30 20:10:47.327526 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:47.327120 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp" podUID="9543233d-9115-4459-87b7-dea47aa8d1ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 30 20:10:48.326486 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:48.326452 4292 patch_prober.go:29] interesting pod/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 30 20:10:48.326650 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:48.326515 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp" podUID="9543233d-9115-4459-87b7-dea47aa8d1ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 30 20:10:49.326312 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:49.326274 4292 patch_prober.go:29] interesting pod/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 30 20:10:49.326715 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:49.326337 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp" podUID="9543233d-9115-4459-87b7-dea47aa8d1ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 30 20:10:50.334613 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:50.334583 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp" Oct 30 20:10:50.335110 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:50.335007 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp" Oct 30 20:10:50.335729 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:50.335711 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp" Oct 30 20:10:52.929490 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:52.929453 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f61bae-kf4l7"] Oct 30 20:10:52.929872 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:52.929859 4292 cpu_manager.go:414] "containerMap: removing container" podUID="f8905c19-f981-4501-8fc1-0bfb9b5520ef" containerName="revoke-keys" Oct 30 20:10:52.929920 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:52.929872 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8905c19-f981-4501-8fc1-0bfb9b5520ef" containerName="revoke-keys" Oct 30 20:10:52.929920 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:52.929886 4292 cpu_manager.go:414] "containerMap: removing container" podUID="931c961c-cdf7-442e-8a6f-ff766e813cad" containerName="istio-proxy" Oct 30 20:10:52.929920 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:52.929891 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="931c961c-cdf7-442e-8a6f-ff766e813cad" containerName="istio-proxy" Oct 30 20:10:52.929920 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:52.929900 4292 cpu_manager.go:414] "containerMap: removing container" podUID="69d11c12-c4cb-452e-90cc-4cbe03333951" containerName="istio-proxy" Oct 30 20:10:52.929920 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:52.929905 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69d11c12-c4cb-452e-90cc-4cbe03333951" containerName="istio-proxy" Oct 30 20:10:52.930061 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:52.929933 4292 cpu_manager.go:414] "containerMap: removing container" podUID="b5978bc5-3ff8-4f4a-b85f-58b737179fee" containerName="istio-proxy" Oct 30 20:10:52.930061 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:52.929938 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5978bc5-3ff8-4f4a-b85f-58b737179fee" containerName="istio-proxy" Oct 30 20:10:52.930061 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:52.930001 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5978bc5-3ff8-4f4a-b85f-58b737179fee" containerName="istio-proxy" Oct 30 20:10:52.930061 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:52.930010 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8905c19-f981-4501-8fc1-0bfb9b5520ef" containerName="revoke-keys" Oct 30 20:10:52.930061 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:52.930016 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="931c961c-cdf7-442e-8a6f-ff766e813cad" containerName="istio-proxy" Oct 30 20:10:52.930061 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:52.930022 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="69d11c12-c4cb-452e-90cc-4cbe03333951" containerName="istio-proxy" Oct 30 20:10:52.933211 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:52.933194 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f61bae-kf4l7" Oct 30 20:10:52.992850 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:52.992809 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f61bae-kf4l7"] Oct 30 20:10:53.037566 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:53.037540 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:10:53.037720 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:53.037591 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\"" Oct 30 20:10:53.056510 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:53.056484 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2271bb43-4cfc-4fe4-8cb2-5e4d6845f34e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f61bae-kf4l7\" (UID: \"2271bb43-4cfc-4fe4-8cb2-5e4d6845f34e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f61bae-kf4l7" Oct 30 20:10:53.056657 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:53.056528 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2271bb43-4cfc-4fe4-8cb2-5e4d6845f34e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f61bae-kf4l7\" (UID: \"2271bb43-4cfc-4fe4-8cb2-5e4d6845f34e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f61bae-kf4l7" Oct 30 20:10:53.056657 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:53.056640 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjkrl\" (UniqueName: \"kubernetes.io/projected/2271bb43-4cfc-4fe4-8cb2-5e4d6845f34e-kube-api-access-bjkrl\") pod \"maas-api-revoke-keys-e2e-praxis-a-f61bae-kf4l7\" (UID: \"2271bb43-4cfc-4fe4-8cb2-5e4d6845f34e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f61bae-kf4l7" Oct 30 20:10:53.158117 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:53.158090 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2271bb43-4cfc-4fe4-8cb2-5e4d6845f34e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f61bae-kf4l7\" (UID: \"2271bb43-4cfc-4fe4-8cb2-5e4d6845f34e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f61bae-kf4l7" Oct 30 20:10:53.158277 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:53.158137 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2271bb43-4cfc-4fe4-8cb2-5e4d6845f34e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f61bae-kf4l7\" (UID: \"2271bb43-4cfc-4fe4-8cb2-5e4d6845f34e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f61bae-kf4l7" Oct 30 20:10:53.158277 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:53.158209 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjkrl\" (UniqueName: \"kubernetes.io/projected/2271bb43-4cfc-4fe4-8cb2-5e4d6845f34e-kube-api-access-bjkrl\") pod \"maas-api-revoke-keys-e2e-praxis-a-f61bae-kf4l7\" (UID: \"2271bb43-4cfc-4fe4-8cb2-5e4d6845f34e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f61bae-kf4l7" Oct 30 20:10:53.158870 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:53.158843 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2271bb43-4cfc-4fe4-8cb2-5e4d6845f34e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f61bae-kf4l7\" (UID: \"2271bb43-4cfc-4fe4-8cb2-5e4d6845f34e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f61bae-kf4l7" Oct 30 20:10:53.160514 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:53.160497 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2271bb43-4cfc-4fe4-8cb2-5e4d6845f34e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f61bae-kf4l7\" (UID: \"2271bb43-4cfc-4fe4-8cb2-5e4d6845f34e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f61bae-kf4l7" Oct 30 20:10:53.221747 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:53.221685 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjkrl\" (UniqueName: \"kubernetes.io/projected/2271bb43-4cfc-4fe4-8cb2-5e4d6845f34e-kube-api-access-bjkrl\") pod \"maas-api-revoke-keys-e2e-praxis-a-f61bae-kf4l7\" (UID: \"2271bb43-4cfc-4fe4-8cb2-5e4d6845f34e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f61bae-kf4l7" Oct 30 20:10:53.245244 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:53.245216 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f61bae-kf4l7" Oct 30 20:10:53.620043 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:53.620016 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f61bae-kf4l7"] Oct 30 20:10:53.622220 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:10:53.622184 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2271bb43_4cfc_4fe4_8cb2_5e4d6845f34e.slice/crio-212e4e0a1b05016492b13f0c82f93ff58a9eac2e4f1cec41cdd42b400277e411 WatchSource:0}: Error finding container 212e4e0a1b05016492b13f0c82f93ff58a9eac2e4f1cec41cdd42b400277e411: Status 404 returned error can't find the container with id 212e4e0a1b05016492b13f0c82f93ff58a9eac2e4f1cec41cdd42b400277e411 Oct 30 20:10:54.250182 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:54.250099 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f61bae-kf4l7" event={"ID":"2271bb43-4cfc-4fe4-8cb2-5e4d6845f34e","Type":"ContainerStarted","Data":"a2c9d8f3c9d550b70e7cb7b2a6deb74c436514662d21a94beec572f7f232c0df"} Oct 30 20:10:54.250182 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:54.250180 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f61bae-kf4l7" event={"ID":"2271bb43-4cfc-4fe4-8cb2-5e4d6845f34e","Type":"ContainerStarted","Data":"212e4e0a1b05016492b13f0c82f93ff58a9eac2e4f1cec41cdd42b400277e411"} Oct 30 20:10:54.370096 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:54.370043 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f61bae-kf4l7" podStartSLOduration=2.370028761 podStartE2EDuration="2.370028761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:10:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:10:54.368769924 +0000 UTC m=+2757.887085392" watchObservedRunningTime="2026-10-30 20:10:54.370028761 +0000 UTC m=+2757.888344246" Oct 30 20:10:55.256697 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:55.256660 4292 generic.go:356] "Generic (PLEG): container finished" podID="2271bb43-4cfc-4fe4-8cb2-5e4d6845f34e" containerID="a2c9d8f3c9d550b70e7cb7b2a6deb74c436514662d21a94beec572f7f232c0df" exitCode=0 Oct 30 20:10:55.256697 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:55.256701 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f61bae-kf4l7" event={"ID":"2271bb43-4cfc-4fe4-8cb2-5e4d6845f34e","Type":"ContainerDied","Data":"a2c9d8f3c9d550b70e7cb7b2a6deb74c436514662d21a94beec572f7f232c0df"} Oct 30 20:10:56.390317 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:56.390288 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f61bae-kf4l7" Oct 30 20:10:56.499555 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:56.499526 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2271bb43-4cfc-4fe4-8cb2-5e4d6845f34e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2271bb43-4cfc-4fe4-8cb2-5e4d6845f34e-maas-api-xks-ca\") pod \"2271bb43-4cfc-4fe4-8cb2-5e4d6845f34e\" (UID: \"2271bb43-4cfc-4fe4-8cb2-5e4d6845f34e\") " Oct 30 20:10:56.499723 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:56.499615 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2271bb43-4cfc-4fe4-8cb2-5e4d6845f34e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2271bb43-4cfc-4fe4-8cb2-5e4d6845f34e-maas-api-openshift-ca\") pod \"2271bb43-4cfc-4fe4-8cb2-5e4d6845f34e\" (UID: \"2271bb43-4cfc-4fe4-8cb2-5e4d6845f34e\") " Oct 30 20:10:56.499723 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:56.499658 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2271bb43-4cfc-4fe4-8cb2-5e4d6845f34e-kube-api-access-bjkrl\" (UniqueName: \"kubernetes.io/projected/2271bb43-4cfc-4fe4-8cb2-5e4d6845f34e-kube-api-access-bjkrl\") pod \"2271bb43-4cfc-4fe4-8cb2-5e4d6845f34e\" (UID: \"2271bb43-4cfc-4fe4-8cb2-5e4d6845f34e\") " Oct 30 20:10:56.500014 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:56.499983 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2271bb43-4cfc-4fe4-8cb2-5e4d6845f34e-maas-api-openshift-ca" pod "2271bb43-4cfc-4fe4-8cb2-5e4d6845f34e" (UID: "2271bb43-4cfc-4fe4-8cb2-5e4d6845f34e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:10:56.501693 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:56.501670 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2271bb43-4cfc-4fe4-8cb2-5e4d6845f34e-maas-api-xks-ca" pod "2271bb43-4cfc-4fe4-8cb2-5e4d6845f34e" (UID: "2271bb43-4cfc-4fe4-8cb2-5e4d6845f34e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:10:56.501787 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:56.501702 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2271bb43-4cfc-4fe4-8cb2-5e4d6845f34e-kube-api-access-bjkrl" pod "2271bb43-4cfc-4fe4-8cb2-5e4d6845f34e" (UID: "2271bb43-4cfc-4fe4-8cb2-5e4d6845f34e"). InnerVolumeSpecName "kube-api-access-bjkrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:10:56.600579 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:56.600550 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjkrl\" (UniqueName: \"kubernetes.io/projected/2271bb43-4cfc-4fe4-8cb2-5e4d6845f34e-kube-api-access-bjkrl\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:56.600579 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:56.600579 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2271bb43-4cfc-4fe4-8cb2-5e4d6845f34e-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:56.600744 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:56.600593 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2271bb43-4cfc-4fe4-8cb2-5e4d6845f34e-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:57.231759 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:57.231731 4292 scope.go:122] "RemoveContainer" containerID="d43d246151fe2246e8b837de664ae2b46622f3e589de4f6dde4a5ac9c789aeda" Oct 30 20:10:57.243416 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:57.243393 4292 scope.go:122] "RemoveContainer" containerID="174dbd00c1252887f88f4e7379f7afc2009bb378ae6b59f164dd943b8830d678" Oct 30 20:10:57.269228 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:57.269207 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f61bae-kf4l7" Oct 30 20:10:57.269340 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:57.269206 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f61bae-kf4l7" event={"ID":"2271bb43-4cfc-4fe4-8cb2-5e4d6845f34e","Type":"ContainerDied","Data":"212e4e0a1b05016492b13f0c82f93ff58a9eac2e4f1cec41cdd42b400277e411"} Oct 30 20:10:57.269434 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:10:57.269414 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="212e4e0a1b05016492b13f0c82f93ff58a9eac2e4f1cec41cdd42b400277e411" Oct 30 20:11:13.221834 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:11:13.221792 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:11:18.284245 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:18.284209 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5"] Oct 30 20:11:18.284706 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:18.284682 4292 cpu_manager.go:414] "containerMap: removing container" podUID="2271bb43-4cfc-4fe4-8cb2-5e4d6845f34e" containerName="revoke-keys" Oct 30 20:11:18.284706 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:18.284695 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2271bb43-4cfc-4fe4-8cb2-5e4d6845f34e" containerName="revoke-keys" Oct 30 20:11:18.284819 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:18.284807 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="2271bb43-4cfc-4fe4-8cb2-5e4d6845f34e" containerName="revoke-keys" Oct 30 20:11:18.289892 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:18.289871 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5" Oct 30 20:11:18.328445 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:18.328391 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6d51da1-a6c7-4060-8da8-74b8b56e949d-workload-certs\") pod \"e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5\" (UID: \"c6d51da1-a6c7-4060-8da8-74b8b56e949d\") " pod="openshift-ingress/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5" Oct 30 20:11:18.328445 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:18.328443 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6d51da1-a6c7-4060-8da8-74b8b56e949d-istio-data\") pod \"e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5\" (UID: \"c6d51da1-a6c7-4060-8da8-74b8b56e949d\") " pod="openshift-ingress/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5" Oct 30 20:11:18.328694 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:18.328512 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6d51da1-a6c7-4060-8da8-74b8b56e949d-workload-socket\") pod \"e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5\" (UID: \"c6d51da1-a6c7-4060-8da8-74b8b56e949d\") " pod="openshift-ingress/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5" Oct 30 20:11:18.328694 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:18.328541 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6d51da1-a6c7-4060-8da8-74b8b56e949d-istio-token\") pod \"e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5\" (UID: \"c6d51da1-a6c7-4060-8da8-74b8b56e949d\") " pod="openshift-ingress/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5" Oct 30 20:11:18.328694 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:18.328572 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq5ft\" (UniqueName: \"kubernetes.io/projected/c6d51da1-a6c7-4060-8da8-74b8b56e949d-kube-api-access-gq5ft\") pod \"e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5\" (UID: \"c6d51da1-a6c7-4060-8da8-74b8b56e949d\") " pod="openshift-ingress/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5" Oct 30 20:11:18.328694 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:18.328645 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6d51da1-a6c7-4060-8da8-74b8b56e949d-credential-socket\") pod \"e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5\" (UID: \"c6d51da1-a6c7-4060-8da8-74b8b56e949d\") " pod="openshift-ingress/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5" Oct 30 20:11:18.328924 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:18.328698 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6d51da1-a6c7-4060-8da8-74b8b56e949d-istio-envoy\") pod \"e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5\" (UID: \"c6d51da1-a6c7-4060-8da8-74b8b56e949d\") " pod="openshift-ingress/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5" Oct 30 20:11:18.328924 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:18.328732 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6d51da1-a6c7-4060-8da8-74b8b56e949d-istiod-ca-cert\") pod \"e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5\" (UID: \"c6d51da1-a6c7-4060-8da8-74b8b56e949d\") " pod="openshift-ingress/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5" Oct 30 20:11:18.328924 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:18.328781 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6d51da1-a6c7-4060-8da8-74b8b56e949d-istio-podinfo\") pod \"e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5\" (UID: \"c6d51da1-a6c7-4060-8da8-74b8b56e949d\") " pod="openshift-ingress/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5" Oct 30 20:11:18.359631 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:18.359598 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-04679c-openshift-default-dockercfg-bf5nd\"" Oct 30 20:11:18.402401 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:18.400827 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5"] Oct 30 20:11:18.429777 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:18.429723 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6d51da1-a6c7-4060-8da8-74b8b56e949d-workload-certs\") pod \"e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5\" (UID: \"c6d51da1-a6c7-4060-8da8-74b8b56e949d\") " pod="openshift-ingress/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5" Oct 30 20:11:18.429962 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:18.429786 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6d51da1-a6c7-4060-8da8-74b8b56e949d-istio-data\") pod \"e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5\" (UID: \"c6d51da1-a6c7-4060-8da8-74b8b56e949d\") " pod="openshift-ingress/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5" Oct 30 20:11:18.429962 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:18.429855 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6d51da1-a6c7-4060-8da8-74b8b56e949d-workload-socket\") pod \"e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5\" (UID: \"c6d51da1-a6c7-4060-8da8-74b8b56e949d\") " pod="openshift-ingress/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5" Oct 30 20:11:18.429962 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:18.429883 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6d51da1-a6c7-4060-8da8-74b8b56e949d-istio-token\") pod \"e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5\" (UID: \"c6d51da1-a6c7-4060-8da8-74b8b56e949d\") " pod="openshift-ingress/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5" Oct 30 20:11:18.429962 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:18.429920 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq5ft\" (UniqueName: \"kubernetes.io/projected/c6d51da1-a6c7-4060-8da8-74b8b56e949d-kube-api-access-gq5ft\") pod \"e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5\" (UID: \"c6d51da1-a6c7-4060-8da8-74b8b56e949d\") " pod="openshift-ingress/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5" Oct 30 20:11:18.429962 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:18.429956 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6d51da1-a6c7-4060-8da8-74b8b56e949d-credential-socket\") pod \"e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5\" (UID: \"c6d51da1-a6c7-4060-8da8-74b8b56e949d\") " pod="openshift-ingress/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5" Oct 30 20:11:18.430216 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:18.430002 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6d51da1-a6c7-4060-8da8-74b8b56e949d-istio-envoy\") pod \"e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5\" (UID: \"c6d51da1-a6c7-4060-8da8-74b8b56e949d\") " pod="openshift-ingress/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5" Oct 30 20:11:18.430216 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:18.430035 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6d51da1-a6c7-4060-8da8-74b8b56e949d-istiod-ca-cert\") pod \"e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5\" (UID: \"c6d51da1-a6c7-4060-8da8-74b8b56e949d\") " pod="openshift-ingress/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5" Oct 30 20:11:18.430216 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:18.430082 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6d51da1-a6c7-4060-8da8-74b8b56e949d-istio-podinfo\") pod \"e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5\" (UID: \"c6d51da1-a6c7-4060-8da8-74b8b56e949d\") " pod="openshift-ingress/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5" Oct 30 20:11:18.430216 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:18.430181 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6d51da1-a6c7-4060-8da8-74b8b56e949d-workload-certs\") pod \"e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5\" (UID: \"c6d51da1-a6c7-4060-8da8-74b8b56e949d\") " pod="openshift-ingress/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5" Oct 30 20:11:18.430461 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:18.430235 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6d51da1-a6c7-4060-8da8-74b8b56e949d-istio-data\") pod \"e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5\" (UID: \"c6d51da1-a6c7-4060-8da8-74b8b56e949d\") " pod="openshift-ingress/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5" Oct 30 20:11:18.430461 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:18.430258 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6d51da1-a6c7-4060-8da8-74b8b56e949d-workload-socket\") pod \"e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5\" (UID: \"c6d51da1-a6c7-4060-8da8-74b8b56e949d\") " pod="openshift-ingress/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5" Oct 30 20:11:18.430672 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:18.430645 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6d51da1-a6c7-4060-8da8-74b8b56e949d-credential-socket\") pod \"e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5\" (UID: \"c6d51da1-a6c7-4060-8da8-74b8b56e949d\") " pod="openshift-ingress/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5" Oct 30 20:11:18.430806 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:18.430763 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6d51da1-a6c7-4060-8da8-74b8b56e949d-istiod-ca-cert\") pod \"e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5\" (UID: \"c6d51da1-a6c7-4060-8da8-74b8b56e949d\") " pod="openshift-ingress/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5" Oct 30 20:11:18.432704 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:18.432676 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6d51da1-a6c7-4060-8da8-74b8b56e949d-istio-envoy\") pod \"e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5\" (UID: \"c6d51da1-a6c7-4060-8da8-74b8b56e949d\") " pod="openshift-ingress/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5" Oct 30 20:11:18.432811 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:18.432795 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6d51da1-a6c7-4060-8da8-74b8b56e949d-istio-podinfo\") pod \"e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5\" (UID: \"c6d51da1-a6c7-4060-8da8-74b8b56e949d\") " pod="openshift-ingress/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5" Oct 30 20:11:18.494703 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:18.494278 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6d51da1-a6c7-4060-8da8-74b8b56e949d-istio-token\") pod \"e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5\" (UID: \"c6d51da1-a6c7-4060-8da8-74b8b56e949d\") " pod="openshift-ingress/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5" Oct 30 20:11:18.494703 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:18.494525 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq5ft\" (UniqueName: \"kubernetes.io/projected/c6d51da1-a6c7-4060-8da8-74b8b56e949d-kube-api-access-gq5ft\") pod \"e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5\" (UID: \"c6d51da1-a6c7-4060-8da8-74b8b56e949d\") " pod="openshift-ingress/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5" Oct 30 20:11:18.602853 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:18.602451 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5" Oct 30 20:11:19.044978 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:19.044951 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5"] Oct 30 20:11:19.047832 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:11:19.047797 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6d51da1_a6c7_4060_8da8_74b8b56e949d.slice/crio-ae0cdf1009986e6339b64ce248e1a738931dc9d60ed3f9bb5755f4c9776aa811 WatchSource:0}: Error finding container ae0cdf1009986e6339b64ce248e1a738931dc9d60ed3f9bb5755f4c9776aa811: Status 404 returned error can't find the container with id ae0cdf1009986e6339b64ce248e1a738931dc9d60ed3f9bb5755f4c9776aa811 Oct 30 20:11:19.050498 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:19.050468 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:11:19.050587 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:19.050533 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:11:19.050587 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:19.050566 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:11:19.402648 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:19.402566 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5" event={"ID":"c6d51da1-a6c7-4060-8da8-74b8b56e949d","Type":"ContainerStarted","Data":"7306e784265f78402f6c9f9a644106ec8cc339a737ad42bf01206674024e8ca2"} Oct 30 20:11:19.402648 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:19.402614 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5" event={"ID":"c6d51da1-a6c7-4060-8da8-74b8b56e949d","Type":"ContainerStarted","Data":"ae0cdf1009986e6339b64ce248e1a738931dc9d60ed3f9bb5755f4c9776aa811"} Oct 30 20:11:19.527783 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:19.527730 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5" podStartSLOduration=1.527713725 podStartE2EDuration="1.527713725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:11:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:11:19.525170977 +0000 UTC m=+2783.043486483" watchObservedRunningTime="2026-10-30 20:11:19.527713725 +0000 UTC m=+2783.046029227" Oct 30 20:11:19.603665 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:19.603632 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5" Oct 30 20:11:20.608785 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:20.608756 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5" Oct 30 20:11:21.414073 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:21.414042 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5" Oct 30 20:11:21.415019 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:21.414999 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5" Oct 30 20:11:23.225579 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:23.225538 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46c6813c-jmwfh"] Oct 30 20:11:23.229428 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:23.229401 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46c6813c-jmwfh" Oct 30 20:11:23.284224 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:23.284191 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46c6813c-jmwfh"] Oct 30 20:11:23.295235 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:23.295204 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpvdg\" (UniqueName: \"kubernetes.io/projected/1f16b42e-55ab-4b16-9450-7a73e5b46a10-kube-api-access-fpvdg\") pod \"maas-api-revoke-sub-46c6813c-jmwfh\" (UID: \"1f16b42e-55ab-4b16-9450-7a73e5b46a10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46c6813c-jmwfh" Oct 30 20:11:23.295398 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:23.295377 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f16b42e-55ab-4b16-9450-7a73e5b46a10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46c6813c-jmwfh\" (UID: \"1f16b42e-55ab-4b16-9450-7a73e5b46a10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46c6813c-jmwfh" Oct 30 20:11:23.295464 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:23.295410 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f16b42e-55ab-4b16-9450-7a73e5b46a10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46c6813c-jmwfh\" (UID: \"1f16b42e-55ab-4b16-9450-7a73e5b46a10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46c6813c-jmwfh" Oct 30 20:11:23.332597 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:23.332568 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:11:23.332749 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:23.332668 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\"" Oct 30 20:11:23.396863 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:23.396833 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpvdg\" (UniqueName: \"kubernetes.io/projected/1f16b42e-55ab-4b16-9450-7a73e5b46a10-kube-api-access-fpvdg\") pod \"maas-api-revoke-sub-46c6813c-jmwfh\" (UID: \"1f16b42e-55ab-4b16-9450-7a73e5b46a10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46c6813c-jmwfh" Oct 30 20:11:23.397034 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:23.396974 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f16b42e-55ab-4b16-9450-7a73e5b46a10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46c6813c-jmwfh\" (UID: \"1f16b42e-55ab-4b16-9450-7a73e5b46a10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46c6813c-jmwfh" Oct 30 20:11:23.397034 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:23.397015 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f16b42e-55ab-4b16-9450-7a73e5b46a10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46c6813c-jmwfh\" (UID: \"1f16b42e-55ab-4b16-9450-7a73e5b46a10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46c6813c-jmwfh" Oct 30 20:11:23.397675 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:23.397648 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f16b42e-55ab-4b16-9450-7a73e5b46a10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46c6813c-jmwfh\" (UID: \"1f16b42e-55ab-4b16-9450-7a73e5b46a10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46c6813c-jmwfh" Oct 30 20:11:23.399360 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:23.399334 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f16b42e-55ab-4b16-9450-7a73e5b46a10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46c6813c-jmwfh\" (UID: \"1f16b42e-55ab-4b16-9450-7a73e5b46a10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46c6813c-jmwfh" Oct 30 20:11:23.469936 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:23.469899 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpvdg\" (UniqueName: \"kubernetes.io/projected/1f16b42e-55ab-4b16-9450-7a73e5b46a10-kube-api-access-fpvdg\") pod \"maas-api-revoke-sub-46c6813c-jmwfh\" (UID: \"1f16b42e-55ab-4b16-9450-7a73e5b46a10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46c6813c-jmwfh" Oct 30 20:11:23.542360 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:23.542332 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46c6813c-jmwfh" Oct 30 20:11:23.917965 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:23.917939 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46c6813c-jmwfh"] Oct 30 20:11:23.920018 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:11:23.919984 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f16b42e_55ab_4b16_9450_7a73e5b46a10.slice/crio-4f1247b3829a5b872894e234ddf7d23c75f0227ca4bde42c59f5bfc1dfa8cdff WatchSource:0}: Error finding container 4f1247b3829a5b872894e234ddf7d23c75f0227ca4bde42c59f5bfc1dfa8cdff: Status 404 returned error can't find the container with id 4f1247b3829a5b872894e234ddf7d23c75f0227ca4bde42c59f5bfc1dfa8cdff Oct 30 20:11:24.430920 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:24.430886 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46c6813c-jmwfh" event={"ID":"1f16b42e-55ab-4b16-9450-7a73e5b46a10","Type":"ContainerStarted","Data":"a493fa2adbba3d6f10099ecdd57ac1e8cbda78618e683657a8a766c070a26216"} Oct 30 20:11:24.431298 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:24.430944 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46c6813c-jmwfh" event={"ID":"1f16b42e-55ab-4b16-9450-7a73e5b46a10","Type":"ContainerStarted","Data":"4f1247b3829a5b872894e234ddf7d23c75f0227ca4bde42c59f5bfc1dfa8cdff"} Oct 30 20:11:24.551165 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:24.551118 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46c6813c-jmwfh" podStartSLOduration=1.5511040459999998 podStartE2EDuration="1.551104046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:11:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:11:24.548866488 +0000 UTC m=+2788.067181945" watchObservedRunningTime="2026-10-30 20:11:24.551104046 +0000 UTC m=+2788.069419500" Oct 30 20:11:25.438007 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:25.437975 4292 generic.go:356] "Generic (PLEG): container finished" podID="1f16b42e-55ab-4b16-9450-7a73e5b46a10" containerID="a493fa2adbba3d6f10099ecdd57ac1e8cbda78618e683657a8a766c070a26216" exitCode=0 Oct 30 20:11:25.438490 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:25.438060 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46c6813c-jmwfh" event={"ID":"1f16b42e-55ab-4b16-9450-7a73e5b46a10","Type":"ContainerDied","Data":"a493fa2adbba3d6f10099ecdd57ac1e8cbda78618e683657a8a766c070a26216"} Oct 30 20:11:26.347623 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:26.347588 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-04679c-696cc7bfd7-zdpv7"] Oct 30 20:11:26.351316 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:26.351296 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-04679c-696cc7bfd7-zdpv7" Oct 30 20:11:26.408280 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:26.408253 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-04679c\"" Oct 30 20:11:26.408647 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:26.408629 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-04679c-dockercfg-6hxr6\"" Oct 30 20:11:26.411370 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:26.411332 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-04679c-696cc7bfd7-zdpv7"] Oct 30 20:11:26.453374 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:26.451262 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dee7e19-6aeb-46a7-b763-52cbd3cf1113-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-04679c-696cc7bfd7-zdpv7\" (UID: \"6dee7e19-6aeb-46a7-b763-52cbd3cf1113\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-04679c-696cc7bfd7-zdpv7" Oct 30 20:11:26.453374 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:26.451383 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs52q\" (UniqueName: \"kubernetes.io/projected/6dee7e19-6aeb-46a7-b763-52cbd3cf1113-kube-api-access-rs52q\") pod \"payload-pre-processing-e2e-praxis-b-04679c-696cc7bfd7-zdpv7\" (UID: \"6dee7e19-6aeb-46a7-b763-52cbd3cf1113\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-04679c-696cc7bfd7-zdpv7" Oct 30 20:11:26.498090 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:26.498040 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-04679c-86c54d7f84-prhpr"] Oct 30 20:11:26.507140 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:26.507107 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-04679c-86c54d7f84-prhpr" Oct 30 20:11:26.516979 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:26.516947 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-04679c-86c54d7f84-prhpr"] Oct 30 20:11:26.552409 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:26.552375 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dee7e19-6aeb-46a7-b763-52cbd3cf1113-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-04679c-696cc7bfd7-zdpv7\" (UID: \"6dee7e19-6aeb-46a7-b763-52cbd3cf1113\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-04679c-696cc7bfd7-zdpv7" Oct 30 20:11:26.552592 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:26.552454 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs52q\" (UniqueName: \"kubernetes.io/projected/6dee7e19-6aeb-46a7-b763-52cbd3cf1113-kube-api-access-rs52q\") pod \"payload-pre-processing-e2e-praxis-b-04679c-696cc7bfd7-zdpv7\" (UID: \"6dee7e19-6aeb-46a7-b763-52cbd3cf1113\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-04679c-696cc7bfd7-zdpv7" Oct 30 20:11:26.553113 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:26.553085 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dee7e19-6aeb-46a7-b763-52cbd3cf1113-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-04679c-696cc7bfd7-zdpv7\" (UID: \"6dee7e19-6aeb-46a7-b763-52cbd3cf1113\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-04679c-696cc7bfd7-zdpv7" Oct 30 20:11:26.616262 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:26.616186 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs52q\" (UniqueName: \"kubernetes.io/projected/6dee7e19-6aeb-46a7-b763-52cbd3cf1113-kube-api-access-rs52q\") pod \"payload-pre-processing-e2e-praxis-b-04679c-696cc7bfd7-zdpv7\" (UID: \"6dee7e19-6aeb-46a7-b763-52cbd3cf1113\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-04679c-696cc7bfd7-zdpv7" Oct 30 20:11:26.653576 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:26.653541 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/191cdc99-91e9-49de-9006-66e40aa246b6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-04679c-86c54d7f84-prhpr\" (UID: \"191cdc99-91e9-49de-9006-66e40aa246b6\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-04679c-86c54d7f84-prhpr" Oct 30 20:11:26.653741 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:26.653714 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj55v\" (UniqueName: \"kubernetes.io/projected/191cdc99-91e9-49de-9006-66e40aa246b6-kube-api-access-bj55v\") pod \"payload-processing-e2e-praxis-b-04679c-86c54d7f84-prhpr\" (UID: \"191cdc99-91e9-49de-9006-66e40aa246b6\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-04679c-86c54d7f84-prhpr" Oct 30 20:11:26.663821 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:26.663796 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-04679c-696cc7bfd7-zdpv7" Oct 30 20:11:26.669085 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:26.669069 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46c6813c-jmwfh" Oct 30 20:11:26.754568 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:26.754297 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f16b42e-55ab-4b16-9450-7a73e5b46a10-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f16b42e-55ab-4b16-9450-7a73e5b46a10-maas-api-openshift-ca\") pod \"1f16b42e-55ab-4b16-9450-7a73e5b46a10\" (UID: \"1f16b42e-55ab-4b16-9450-7a73e5b46a10\") " Oct 30 20:11:26.754568 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:26.754456 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f16b42e-55ab-4b16-9450-7a73e5b46a10-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f16b42e-55ab-4b16-9450-7a73e5b46a10-maas-api-xks-ca\") pod \"1f16b42e-55ab-4b16-9450-7a73e5b46a10\" (UID: \"1f16b42e-55ab-4b16-9450-7a73e5b46a10\") " Oct 30 20:11:26.754568 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:26.754567 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f16b42e-55ab-4b16-9450-7a73e5b46a10-kube-api-access-fpvdg\" (UniqueName: \"kubernetes.io/projected/1f16b42e-55ab-4b16-9450-7a73e5b46a10-kube-api-access-fpvdg\") pod \"1f16b42e-55ab-4b16-9450-7a73e5b46a10\" (UID: \"1f16b42e-55ab-4b16-9450-7a73e5b46a10\") " Oct 30 20:11:26.756457 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:26.754936 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/191cdc99-91e9-49de-9006-66e40aa246b6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-04679c-86c54d7f84-prhpr\" (UID: \"191cdc99-91e9-49de-9006-66e40aa246b6\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-04679c-86c54d7f84-prhpr" Oct 30 20:11:26.756457 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:26.755058 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj55v\" (UniqueName: \"kubernetes.io/projected/191cdc99-91e9-49de-9006-66e40aa246b6-kube-api-access-bj55v\") pod \"payload-processing-e2e-praxis-b-04679c-86c54d7f84-prhpr\" (UID: \"191cdc99-91e9-49de-9006-66e40aa246b6\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-04679c-86c54d7f84-prhpr" Oct 30 20:11:26.756457 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:26.755754 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f16b42e-55ab-4b16-9450-7a73e5b46a10-maas-api-openshift-ca" pod "1f16b42e-55ab-4b16-9450-7a73e5b46a10" (UID: "1f16b42e-55ab-4b16-9450-7a73e5b46a10"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:11:26.758172 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:26.758145 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/191cdc99-91e9-49de-9006-66e40aa246b6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-04679c-86c54d7f84-prhpr\" (UID: \"191cdc99-91e9-49de-9006-66e40aa246b6\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-04679c-86c54d7f84-prhpr" Oct 30 20:11:26.758952 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:26.758912 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f16b42e-55ab-4b16-9450-7a73e5b46a10-kube-api-access-fpvdg" pod "1f16b42e-55ab-4b16-9450-7a73e5b46a10" (UID: "1f16b42e-55ab-4b16-9450-7a73e5b46a10"). InnerVolumeSpecName "kube-api-access-fpvdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:11:26.761730 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:26.761705 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f16b42e-55ab-4b16-9450-7a73e5b46a10-maas-api-xks-ca" pod "1f16b42e-55ab-4b16-9450-7a73e5b46a10" (UID: "1f16b42e-55ab-4b16-9450-7a73e5b46a10"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:11:26.817804 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:26.817769 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj55v\" (UniqueName: \"kubernetes.io/projected/191cdc99-91e9-49de-9006-66e40aa246b6-kube-api-access-bj55v\") pod \"payload-processing-e2e-praxis-b-04679c-86c54d7f84-prhpr\" (UID: \"191cdc99-91e9-49de-9006-66e40aa246b6\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-04679c-86c54d7f84-prhpr" Oct 30 20:11:26.831606 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:26.831580 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-04679c-86c54d7f84-prhpr" Oct 30 20:11:26.856267 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:26.856242 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpvdg\" (UniqueName: \"kubernetes.io/projected/1f16b42e-55ab-4b16-9450-7a73e5b46a10-kube-api-access-fpvdg\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:26.856267 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:26.856266 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f16b42e-55ab-4b16-9450-7a73e5b46a10-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:26.856432 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:26.856278 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f16b42e-55ab-4b16-9450-7a73e5b46a10-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:27.041726 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:27.041695 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-04679c-696cc7bfd7-zdpv7"] Oct 30 20:11:27.043576 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:11:27.043541 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dee7e19_6aeb_46a7_b763_52cbd3cf1113.slice/crio-97212bdc0cbce7b389dfb5093eb79a621e52888367481bc54261d9bb831946d2 WatchSource:0}: Error finding container 97212bdc0cbce7b389dfb5093eb79a621e52888367481bc54261d9bb831946d2: Status 404 returned error can't find the container with id 97212bdc0cbce7b389dfb5093eb79a621e52888367481bc54261d9bb831946d2 Oct 30 20:11:27.209212 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:27.209185 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-04679c-86c54d7f84-prhpr"] Oct 30 20:11:27.210807 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:11:27.210779 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod191cdc99_91e9_49de_9006_66e40aa246b6.slice/crio-0efef310c6440fc5e545adf93688a30d4efa9e64aadc06eb544eedd5d152c078 WatchSource:0}: Error finding container 0efef310c6440fc5e545adf93688a30d4efa9e64aadc06eb544eedd5d152c078: Status 404 returned error can't find the container with id 0efef310c6440fc5e545adf93688a30d4efa9e64aadc06eb544eedd5d152c078 Oct 30 20:11:27.506219 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:27.506113 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-04679c-86c54d7f84-prhpr" event={"ID":"191cdc99-91e9-49de-9006-66e40aa246b6","Type":"ContainerStarted","Data":"96dd42c675b063c1f91ab2385392830cdaf9832d9e83ebefc10981e5640605df"} Oct 30 20:11:27.506219 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:27.506171 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-04679c-86c54d7f84-prhpr" event={"ID":"191cdc99-91e9-49de-9006-66e40aa246b6","Type":"ContainerStarted","Data":"0efef310c6440fc5e545adf93688a30d4efa9e64aadc06eb544eedd5d152c078"} Oct 30 20:11:27.506219 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:27.506201 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-04679c-86c54d7f84-prhpr" Oct 30 20:11:27.507601 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:27.507578 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-04679c-696cc7bfd7-zdpv7" event={"ID":"6dee7e19-6aeb-46a7-b763-52cbd3cf1113","Type":"ContainerStarted","Data":"3dbf1983887a6ed07df40468dbfd9096f7b13f2b7dba14318e7a86f03a6a7025"} Oct 30 20:11:27.507709 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:27.507615 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-04679c-696cc7bfd7-zdpv7" event={"ID":"6dee7e19-6aeb-46a7-b763-52cbd3cf1113","Type":"ContainerStarted","Data":"97212bdc0cbce7b389dfb5093eb79a621e52888367481bc54261d9bb831946d2"} Oct 30 20:11:27.507709 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:27.507670 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-04679c-696cc7bfd7-zdpv7" Oct 30 20:11:27.508890 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:27.508869 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46c6813c-jmwfh" event={"ID":"1f16b42e-55ab-4b16-9450-7a73e5b46a10","Type":"ContainerDied","Data":"4f1247b3829a5b872894e234ddf7d23c75f0227ca4bde42c59f5bfc1dfa8cdff"} Oct 30 20:11:27.508890 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:27.508881 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46c6813c-jmwfh" Oct 30 20:11:27.509032 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:27.508903 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f1247b3829a5b872894e234ddf7d23c75f0227ca4bde42c59f5bfc1dfa8cdff" Oct 30 20:11:27.635400 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:27.635335 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-04679c-86c54d7f84-prhpr" podStartSLOduration=1.635322975 podStartE2EDuration="1.635322975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:11:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:11:27.632089063 +0000 UTC m=+2791.150404519" watchObservedRunningTime="2026-10-30 20:11:27.635322975 +0000 UTC m=+2791.153638463" Oct 30 20:11:27.761260 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:27.757553 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-04679c-696cc7bfd7-zdpv7" podStartSLOduration=1.75753492 podStartE2EDuration="1.75753492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:11:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:11:27.754737839 +0000 UTC m=+2791.273053311" watchObservedRunningTime="2026-10-30 20:11:27.75753492 +0000 UTC m=+2791.275850379" Oct 30 20:11:36.203918 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:36.203884 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-84ad63-dvdq8"] Oct 30 20:11:36.204346 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:36.204333 4292 cpu_manager.go:414] "containerMap: removing container" podUID="1f16b42e-55ab-4b16-9450-7a73e5b46a10" containerName="revoke-keys" Oct 30 20:11:36.204410 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:36.204373 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f16b42e-55ab-4b16-9450-7a73e5b46a10" containerName="revoke-keys" Oct 30 20:11:36.204458 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:36.204453 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f16b42e-55ab-4b16-9450-7a73e5b46a10" containerName="revoke-keys" Oct 30 20:11:36.207839 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:36.207823 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-84ad63-dvdq8" Oct 30 20:11:36.262500 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:36.262472 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-84ad63-dvdq8"] Oct 30 20:11:36.291096 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:36.291063 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf7fc\" (UniqueName: \"kubernetes.io/projected/26d6a553-565d-4095-b600-ea96b0bb46bd-kube-api-access-gf7fc\") pod \"maas-api-revoke-keys-e2e-praxis-a-84ad63-dvdq8\" (UID: \"26d6a553-565d-4095-b600-ea96b0bb46bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-84ad63-dvdq8" Oct 30 20:11:36.291274 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:36.291124 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26d6a553-565d-4095-b600-ea96b0bb46bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-84ad63-dvdq8\" (UID: \"26d6a553-565d-4095-b600-ea96b0bb46bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-84ad63-dvdq8" Oct 30 20:11:36.291367 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:36.291271 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26d6a553-565d-4095-b600-ea96b0bb46bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-84ad63-dvdq8\" (UID: \"26d6a553-565d-4095-b600-ea96b0bb46bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-84ad63-dvdq8" Oct 30 20:11:36.309413 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:36.309384 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:11:36.309601 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:36.309589 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\"" Oct 30 20:11:36.392284 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:36.392247 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26d6a553-565d-4095-b600-ea96b0bb46bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-84ad63-dvdq8\" (UID: \"26d6a553-565d-4095-b600-ea96b0bb46bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-84ad63-dvdq8" Oct 30 20:11:36.392474 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:36.392401 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf7fc\" (UniqueName: \"kubernetes.io/projected/26d6a553-565d-4095-b600-ea96b0bb46bd-kube-api-access-gf7fc\") pod \"maas-api-revoke-keys-e2e-praxis-a-84ad63-dvdq8\" (UID: \"26d6a553-565d-4095-b600-ea96b0bb46bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-84ad63-dvdq8" Oct 30 20:11:36.392474 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:36.392437 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26d6a553-565d-4095-b600-ea96b0bb46bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-84ad63-dvdq8\" (UID: \"26d6a553-565d-4095-b600-ea96b0bb46bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-84ad63-dvdq8" Oct 30 20:11:36.393057 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:36.393024 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26d6a553-565d-4095-b600-ea96b0bb46bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-84ad63-dvdq8\" (UID: \"26d6a553-565d-4095-b600-ea96b0bb46bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-84ad63-dvdq8" Oct 30 20:11:36.394675 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:36.394647 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26d6a553-565d-4095-b600-ea96b0bb46bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-84ad63-dvdq8\" (UID: \"26d6a553-565d-4095-b600-ea96b0bb46bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-84ad63-dvdq8" Oct 30 20:11:36.454911 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:36.454850 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf7fc\" (UniqueName: \"kubernetes.io/projected/26d6a553-565d-4095-b600-ea96b0bb46bd-kube-api-access-gf7fc\") pod \"maas-api-revoke-keys-e2e-praxis-a-84ad63-dvdq8\" (UID: \"26d6a553-565d-4095-b600-ea96b0bb46bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-84ad63-dvdq8" Oct 30 20:11:36.520628 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:36.520602 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-84ad63-dvdq8" Oct 30 20:11:36.915750 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:36.915726 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-84ad63-dvdq8"] Oct 30 20:11:36.917955 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:11:36.917929 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26d6a553_565d_4095_b600_ea96b0bb46bd.slice/crio-1e1bb45172426e67aa9a186de34dedfb0b725e917b374788416dcb3a63921818 WatchSource:0}: Error finding container 1e1bb45172426e67aa9a186de34dedfb0b725e917b374788416dcb3a63921818: Status 404 returned error can't find the container with id 1e1bb45172426e67aa9a186de34dedfb0b725e917b374788416dcb3a63921818 Oct 30 20:11:37.565517 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:37.565453 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-84ad63-dvdq8" event={"ID":"26d6a553-565d-4095-b600-ea96b0bb46bd","Type":"ContainerStarted","Data":"7481fbbc97dc3ae86fac00af8a1b639ba9b1c85047a6cd1f9f76fef350c04028"} Oct 30 20:11:37.565517 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:37.565505 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-84ad63-dvdq8" event={"ID":"26d6a553-565d-4095-b600-ea96b0bb46bd","Type":"ContainerStarted","Data":"1e1bb45172426e67aa9a186de34dedfb0b725e917b374788416dcb3a63921818"} Oct 30 20:11:37.686106 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:37.686053 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-84ad63-dvdq8" podStartSLOduration=1.686039521 podStartE2EDuration="1.686039521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:11:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:11:37.682809534 +0000 UTC m=+2801.201124992" watchObservedRunningTime="2026-10-30 20:11:37.686039521 +0000 UTC m=+2801.204354999" Oct 30 20:11:38.517154 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:38.517121 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-04679c-86c54d7f84-prhpr" Oct 30 20:11:38.517322 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:38.517189 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-04679c-696cc7bfd7-zdpv7" Oct 30 20:11:38.573344 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:38.573299 4292 generic.go:356] "Generic (PLEG): container finished" podID="26d6a553-565d-4095-b600-ea96b0bb46bd" containerID="7481fbbc97dc3ae86fac00af8a1b639ba9b1c85047a6cd1f9f76fef350c04028" exitCode=0 Oct 30 20:11:38.573731 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:38.573394 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-84ad63-dvdq8" event={"ID":"26d6a553-565d-4095-b600-ea96b0bb46bd","Type":"ContainerDied","Data":"7481fbbc97dc3ae86fac00af8a1b639ba9b1c85047a6cd1f9f76fef350c04028"} Oct 30 20:11:39.724738 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:39.724715 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-84ad63-dvdq8" Oct 30 20:11:39.829993 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:39.829918 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26d6a553-565d-4095-b600-ea96b0bb46bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26d6a553-565d-4095-b600-ea96b0bb46bd-maas-api-xks-ca\") pod \"26d6a553-565d-4095-b600-ea96b0bb46bd\" (UID: \"26d6a553-565d-4095-b600-ea96b0bb46bd\") " Oct 30 20:11:39.829993 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:39.829978 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26d6a553-565d-4095-b600-ea96b0bb46bd-kube-api-access-gf7fc\" (UniqueName: \"kubernetes.io/projected/26d6a553-565d-4095-b600-ea96b0bb46bd-kube-api-access-gf7fc\") pod \"26d6a553-565d-4095-b600-ea96b0bb46bd\" (UID: \"26d6a553-565d-4095-b600-ea96b0bb46bd\") " Oct 30 20:11:39.830221 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:39.830016 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26d6a553-565d-4095-b600-ea96b0bb46bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26d6a553-565d-4095-b600-ea96b0bb46bd-maas-api-openshift-ca\") pod \"26d6a553-565d-4095-b600-ea96b0bb46bd\" (UID: \"26d6a553-565d-4095-b600-ea96b0bb46bd\") " Oct 30 20:11:39.830422 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:39.830395 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26d6a553-565d-4095-b600-ea96b0bb46bd-maas-api-openshift-ca" pod "26d6a553-565d-4095-b600-ea96b0bb46bd" (UID: "26d6a553-565d-4095-b600-ea96b0bb46bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:11:39.831945 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:39.831915 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26d6a553-565d-4095-b600-ea96b0bb46bd-maas-api-xks-ca" pod "26d6a553-565d-4095-b600-ea96b0bb46bd" (UID: "26d6a553-565d-4095-b600-ea96b0bb46bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:11:39.832035 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:39.831996 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26d6a553-565d-4095-b600-ea96b0bb46bd-kube-api-access-gf7fc" pod "26d6a553-565d-4095-b600-ea96b0bb46bd" (UID: "26d6a553-565d-4095-b600-ea96b0bb46bd"). InnerVolumeSpecName "kube-api-access-gf7fc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:11:39.931909 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:39.931876 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26d6a553-565d-4095-b600-ea96b0bb46bd-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:39.931909 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:39.931906 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf7fc\" (UniqueName: \"kubernetes.io/projected/26d6a553-565d-4095-b600-ea96b0bb46bd-kube-api-access-gf7fc\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:39.931909 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:39.931917 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26d6a553-565d-4095-b600-ea96b0bb46bd-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:40.587711 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:40.587676 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-84ad63-dvdq8" Oct 30 20:11:40.587882 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:40.587674 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-84ad63-dvdq8" event={"ID":"26d6a553-565d-4095-b600-ea96b0bb46bd","Type":"ContainerDied","Data":"1e1bb45172426e67aa9a186de34dedfb0b725e917b374788416dcb3a63921818"} Oct 30 20:11:40.587882 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:40.587816 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e1bb45172426e67aa9a186de34dedfb0b725e917b374788416dcb3a63921818" Oct 30 20:11:42.701403 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:42.701363 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f61bae-kf4l7"] Oct 30 20:11:42.747012 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:42.746985 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f61bae-kf4l7"] Oct 30 20:11:43.220388 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:11:43.220325 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:11:44.584600 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:44.584559 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2271bb43-4cfc-4fe4-8cb2-5e4d6845f34e" path="/var/lib/kubelet/pods/2271bb43-4cfc-4fe4-8cb2-5e4d6845f34e/volumes" Oct 30 20:11:48.537799 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:48.537756 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-389c4e/praxis-route-389c4e-kserve-7788b879cc-585s7"] Oct 30 20:11:48.538204 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:48.538184 4292 cpu_manager.go:414] "containerMap: removing container" podUID="26d6a553-565d-4095-b600-ea96b0bb46bd" containerName="revoke-keys" Oct 30 20:11:48.538204 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:48.538195 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26d6a553-565d-4095-b600-ea96b0bb46bd" containerName="revoke-keys" Oct 30 20:11:48.538291 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:48.538282 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="26d6a553-565d-4095-b600-ea96b0bb46bd" containerName="revoke-keys" Oct 30 20:11:48.542543 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:48.542526 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-389c4e/praxis-route-389c4e-kserve-7788b879cc-585s7" Oct 30 20:11:48.597484 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:48.597454 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-389c4e/praxis-route-389c4e-kserve-7788b879cc-585s7"] Oct 30 20:11:48.641104 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:48.641072 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55a33bbd-4133-455e-8757-6cd6c6e229c2-dshm\") pod \"praxis-route-389c4e-kserve-7788b879cc-585s7\" (UID: \"55a33bbd-4133-455e-8757-6cd6c6e229c2\") " pod="ai-tenant-e2e-praxis-a-389c4e/praxis-route-389c4e-kserve-7788b879cc-585s7" Oct 30 20:11:48.641317 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:48.641127 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55a33bbd-4133-455e-8757-6cd6c6e229c2-tmp-dir\") pod \"praxis-route-389c4e-kserve-7788b879cc-585s7\" (UID: \"55a33bbd-4133-455e-8757-6cd6c6e229c2\") " pod="ai-tenant-e2e-praxis-a-389c4e/praxis-route-389c4e-kserve-7788b879cc-585s7" Oct 30 20:11:48.641317 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:48.641259 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55a33bbd-4133-455e-8757-6cd6c6e229c2-model-cache\") pod \"praxis-route-389c4e-kserve-7788b879cc-585s7\" (UID: \"55a33bbd-4133-455e-8757-6cd6c6e229c2\") " pod="ai-tenant-e2e-praxis-a-389c4e/praxis-route-389c4e-kserve-7788b879cc-585s7" Oct 30 20:11:48.641481 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:48.641326 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76n7m\" (UniqueName: \"kubernetes.io/projected/55a33bbd-4133-455e-8757-6cd6c6e229c2-kube-api-access-76n7m\") pod \"praxis-route-389c4e-kserve-7788b879cc-585s7\" (UID: \"55a33bbd-4133-455e-8757-6cd6c6e229c2\") " pod="ai-tenant-e2e-praxis-a-389c4e/praxis-route-389c4e-kserve-7788b879cc-585s7" Oct 30 20:11:48.641481 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:48.641412 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55a33bbd-4133-455e-8757-6cd6c6e229c2-tls-certs\") pod \"praxis-route-389c4e-kserve-7788b879cc-585s7\" (UID: \"55a33bbd-4133-455e-8757-6cd6c6e229c2\") " pod="ai-tenant-e2e-praxis-a-389c4e/praxis-route-389c4e-kserve-7788b879cc-585s7" Oct 30 20:11:48.641591 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:48.641520 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55a33bbd-4133-455e-8757-6cd6c6e229c2-home\") pod \"praxis-route-389c4e-kserve-7788b879cc-585s7\" (UID: \"55a33bbd-4133-455e-8757-6cd6c6e229c2\") " pod="ai-tenant-e2e-praxis-a-389c4e/praxis-route-389c4e-kserve-7788b879cc-585s7" Oct 30 20:11:48.645737 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:48.645710 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-389c4e\"/\"openshift-service-ca.crt\"" Oct 30 20:11:48.645852 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:48.645775 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-389c4e\"/\"kube-root-ca.crt\"" Oct 30 20:11:48.646548 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:48.646535 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-389c4e\"/\"default-dockercfg-4rr29\"" Oct 30 20:11:48.646632 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:48.646545 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-389c4e\"/\"praxis-route-389c4e-kserve-self-signed-certs\"" Oct 30 20:11:48.744297 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:48.744253 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55a33bbd-4133-455e-8757-6cd6c6e229c2-dshm\") pod \"praxis-route-389c4e-kserve-7788b879cc-585s7\" (UID: \"55a33bbd-4133-455e-8757-6cd6c6e229c2\") " pod="ai-tenant-e2e-praxis-a-389c4e/praxis-route-389c4e-kserve-7788b879cc-585s7" Oct 30 20:11:48.744630 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:48.744602 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55a33bbd-4133-455e-8757-6cd6c6e229c2-tmp-dir\") pod \"praxis-route-389c4e-kserve-7788b879cc-585s7\" (UID: \"55a33bbd-4133-455e-8757-6cd6c6e229c2\") " pod="ai-tenant-e2e-praxis-a-389c4e/praxis-route-389c4e-kserve-7788b879cc-585s7" Oct 30 20:11:48.744835 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:48.744815 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55a33bbd-4133-455e-8757-6cd6c6e229c2-model-cache\") pod \"praxis-route-389c4e-kserve-7788b879cc-585s7\" (UID: \"55a33bbd-4133-455e-8757-6cd6c6e229c2\") " pod="ai-tenant-e2e-praxis-a-389c4e/praxis-route-389c4e-kserve-7788b879cc-585s7" Oct 30 20:11:48.744938 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:48.744857 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76n7m\" (UniqueName: \"kubernetes.io/projected/55a33bbd-4133-455e-8757-6cd6c6e229c2-kube-api-access-76n7m\") pod \"praxis-route-389c4e-kserve-7788b879cc-585s7\" (UID: \"55a33bbd-4133-455e-8757-6cd6c6e229c2\") " pod="ai-tenant-e2e-praxis-a-389c4e/praxis-route-389c4e-kserve-7788b879cc-585s7" Oct 30 20:11:48.744938 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:48.744905 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55a33bbd-4133-455e-8757-6cd6c6e229c2-tls-certs\") pod \"praxis-route-389c4e-kserve-7788b879cc-585s7\" (UID: \"55a33bbd-4133-455e-8757-6cd6c6e229c2\") " pod="ai-tenant-e2e-praxis-a-389c4e/praxis-route-389c4e-kserve-7788b879cc-585s7" Oct 30 20:11:48.745045 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:48.744963 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55a33bbd-4133-455e-8757-6cd6c6e229c2-home\") pod \"praxis-route-389c4e-kserve-7788b879cc-585s7\" (UID: \"55a33bbd-4133-455e-8757-6cd6c6e229c2\") " pod="ai-tenant-e2e-praxis-a-389c4e/praxis-route-389c4e-kserve-7788b879cc-585s7" Oct 30 20:11:48.745541 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:48.745516 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55a33bbd-4133-455e-8757-6cd6c6e229c2-home\") pod \"praxis-route-389c4e-kserve-7788b879cc-585s7\" (UID: \"55a33bbd-4133-455e-8757-6cd6c6e229c2\") " pod="ai-tenant-e2e-praxis-a-389c4e/praxis-route-389c4e-kserve-7788b879cc-585s7" Oct 30 20:11:48.745815 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:48.745787 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55a33bbd-4133-455e-8757-6cd6c6e229c2-model-cache\") pod \"praxis-route-389c4e-kserve-7788b879cc-585s7\" (UID: \"55a33bbd-4133-455e-8757-6cd6c6e229c2\") " pod="ai-tenant-e2e-praxis-a-389c4e/praxis-route-389c4e-kserve-7788b879cc-585s7" Oct 30 20:11:48.746623 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:48.746596 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55a33bbd-4133-455e-8757-6cd6c6e229c2-tmp-dir\") pod \"praxis-route-389c4e-kserve-7788b879cc-585s7\" (UID: \"55a33bbd-4133-455e-8757-6cd6c6e229c2\") " pod="ai-tenant-e2e-praxis-a-389c4e/praxis-route-389c4e-kserve-7788b879cc-585s7" Oct 30 20:11:48.749723 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:48.749694 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55a33bbd-4133-455e-8757-6cd6c6e229c2-tls-certs\") pod \"praxis-route-389c4e-kserve-7788b879cc-585s7\" (UID: \"55a33bbd-4133-455e-8757-6cd6c6e229c2\") " pod="ai-tenant-e2e-praxis-a-389c4e/praxis-route-389c4e-kserve-7788b879cc-585s7" Oct 30 20:11:48.750228 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:48.750200 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55a33bbd-4133-455e-8757-6cd6c6e229c2-dshm\") pod \"praxis-route-389c4e-kserve-7788b879cc-585s7\" (UID: \"55a33bbd-4133-455e-8757-6cd6c6e229c2\") " pod="ai-tenant-e2e-praxis-a-389c4e/praxis-route-389c4e-kserve-7788b879cc-585s7" Oct 30 20:11:48.824613 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:48.824543 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76n7m\" (UniqueName: \"kubernetes.io/projected/55a33bbd-4133-455e-8757-6cd6c6e229c2-kube-api-access-76n7m\") pod \"praxis-route-389c4e-kserve-7788b879cc-585s7\" (UID: \"55a33bbd-4133-455e-8757-6cd6c6e229c2\") " pod="ai-tenant-e2e-praxis-a-389c4e/praxis-route-389c4e-kserve-7788b879cc-585s7" Oct 30 20:11:48.855477 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:48.855445 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-389c4e/praxis-route-389c4e-kserve-7788b879cc-585s7" Oct 30 20:11:49.262486 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:49.260222 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-389c4e/praxis-route-389c4e-kserve-7788b879cc-585s7"] Oct 30 20:11:49.643661 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:49.643620 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-389c4e/praxis-route-389c4e-kserve-7788b879cc-585s7" event={"ID":"55a33bbd-4133-455e-8757-6cd6c6e229c2","Type":"ContainerStarted","Data":"5c547e0107ea67ba56327332f496f9074281f01cfcb5721e43a46c7ce1a9c51b"} Oct 30 20:11:49.644141 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:49.644112 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-389c4e/praxis-route-389c4e-kserve-7788b879cc-585s7" event={"ID":"55a33bbd-4133-455e-8757-6cd6c6e229c2","Type":"ContainerStarted","Data":"9ae48489e1704d6cf93358b2227ee4c632e2f02a86da3be00b2078f009d332a5"} Oct 30 20:11:49.766404 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:49.766338 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-389c4e/praxis-route-389c4e-kserve-7788b879cc-585s7" podStartSLOduration=1.766325583 podStartE2EDuration="1.766325583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:11:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:11:49.763597884 +0000 UTC m=+2813.281913342" watchObservedRunningTime="2026-10-30 20:11:49.766325583 +0000 UTC m=+2813.284641039" Oct 30 20:11:49.855558 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:49.855522 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-389c4e/praxis-route-389c4e-kserve-7788b879cc-585s7" Oct 30 20:11:51.510806 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:51.510771 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-115a78-599ch"] Oct 30 20:11:51.514477 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:51.514460 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-115a78-599ch" Oct 30 20:11:51.568801 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:51.568768 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-115a78-599ch"] Oct 30 20:11:51.616783 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:51.616749 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:11:51.616921 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:51.616830 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\"" Oct 30 20:11:51.692061 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:51.692025 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9d684e2-416f-4a45-b35b-e7666e064fd4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-115a78-599ch\" (UID: \"a9d684e2-416f-4a45-b35b-e7666e064fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-115a78-599ch" Oct 30 20:11:51.692252 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:51.692233 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndjcx\" (UniqueName: \"kubernetes.io/projected/a9d684e2-416f-4a45-b35b-e7666e064fd4-kube-api-access-ndjcx\") pod \"maas-api-revoke-keys-e2e-praxis-b-115a78-599ch\" (UID: \"a9d684e2-416f-4a45-b35b-e7666e064fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-115a78-599ch" Oct 30 20:11:51.692311 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:51.692272 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9d684e2-416f-4a45-b35b-e7666e064fd4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-115a78-599ch\" (UID: \"a9d684e2-416f-4a45-b35b-e7666e064fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-115a78-599ch" Oct 30 20:11:51.792855 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:51.792823 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9d684e2-416f-4a45-b35b-e7666e064fd4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-115a78-599ch\" (UID: \"a9d684e2-416f-4a45-b35b-e7666e064fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-115a78-599ch" Oct 30 20:11:51.793048 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:51.793030 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndjcx\" (UniqueName: \"kubernetes.io/projected/a9d684e2-416f-4a45-b35b-e7666e064fd4-kube-api-access-ndjcx\") pod \"maas-api-revoke-keys-e2e-praxis-b-115a78-599ch\" (UID: \"a9d684e2-416f-4a45-b35b-e7666e064fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-115a78-599ch" Oct 30 20:11:51.793130 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:51.793069 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9d684e2-416f-4a45-b35b-e7666e064fd4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-115a78-599ch\" (UID: \"a9d684e2-416f-4a45-b35b-e7666e064fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-115a78-599ch" Oct 30 20:11:51.793721 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:51.793700 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9d684e2-416f-4a45-b35b-e7666e064fd4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-115a78-599ch\" (UID: \"a9d684e2-416f-4a45-b35b-e7666e064fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-115a78-599ch" Oct 30 20:11:51.795289 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:51.795267 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9d684e2-416f-4a45-b35b-e7666e064fd4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-115a78-599ch\" (UID: \"a9d684e2-416f-4a45-b35b-e7666e064fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-115a78-599ch" Oct 30 20:11:51.855197 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:51.855172 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndjcx\" (UniqueName: \"kubernetes.io/projected/a9d684e2-416f-4a45-b35b-e7666e064fd4-kube-api-access-ndjcx\") pod \"maas-api-revoke-keys-e2e-praxis-b-115a78-599ch\" (UID: \"a9d684e2-416f-4a45-b35b-e7666e064fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-115a78-599ch" Oct 30 20:11:52.128770 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:52.128698 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-115a78-599ch" Oct 30 20:11:52.506979 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:52.506901 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-115a78-599ch"] Oct 30 20:11:52.510032 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:11:52.510006 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9d684e2_416f_4a45_b35b_e7666e064fd4.slice/crio-342559faef7315d22827bb1b6e90a67ca7f453d8c982fe94a74c3b6cc1e7cd51 WatchSource:0}: Error finding container 342559faef7315d22827bb1b6e90a67ca7f453d8c982fe94a74c3b6cc1e7cd51: Status 404 returned error can't find the container with id 342559faef7315d22827bb1b6e90a67ca7f453d8c982fe94a74c3b6cc1e7cd51 Oct 30 20:11:52.659882 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:52.659853 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-115a78-599ch" event={"ID":"a9d684e2-416f-4a45-b35b-e7666e064fd4","Type":"ContainerStarted","Data":"342559faef7315d22827bb1b6e90a67ca7f453d8c982fe94a74c3b6cc1e7cd51"} Oct 30 20:11:53.667128 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:53.667095 4292 generic.go:356] "Generic (PLEG): container finished" podID="a9d684e2-416f-4a45-b35b-e7666e064fd4" containerID="469be6aa1928edddbba290c1c3236739608aa4a4b337049097da895e74498039" exitCode=0 Oct 30 20:11:53.667582 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:53.667153 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-115a78-599ch" event={"ID":"a9d684e2-416f-4a45-b35b-e7666e064fd4","Type":"ContainerDied","Data":"469be6aa1928edddbba290c1c3236739608aa4a4b337049097da895e74498039"} Oct 30 20:11:54.825390 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:54.825341 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-115a78-599ch" Oct 30 20:11:54.929406 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:54.929377 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9d684e2-416f-4a45-b35b-e7666e064fd4-kube-api-access-ndjcx\" (UniqueName: \"kubernetes.io/projected/a9d684e2-416f-4a45-b35b-e7666e064fd4-kube-api-access-ndjcx\") pod \"a9d684e2-416f-4a45-b35b-e7666e064fd4\" (UID: \"a9d684e2-416f-4a45-b35b-e7666e064fd4\") " Oct 30 20:11:54.929589 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:54.929423 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9d684e2-416f-4a45-b35b-e7666e064fd4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9d684e2-416f-4a45-b35b-e7666e064fd4-maas-api-xks-ca\") pod \"a9d684e2-416f-4a45-b35b-e7666e064fd4\" (UID: \"a9d684e2-416f-4a45-b35b-e7666e064fd4\") " Oct 30 20:11:54.929589 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:54.929456 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9d684e2-416f-4a45-b35b-e7666e064fd4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9d684e2-416f-4a45-b35b-e7666e064fd4-maas-api-openshift-ca\") pod \"a9d684e2-416f-4a45-b35b-e7666e064fd4\" (UID: \"a9d684e2-416f-4a45-b35b-e7666e064fd4\") " Oct 30 20:11:54.929879 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:54.929842 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9d684e2-416f-4a45-b35b-e7666e064fd4-maas-api-openshift-ca" pod "a9d684e2-416f-4a45-b35b-e7666e064fd4" (UID: "a9d684e2-416f-4a45-b35b-e7666e064fd4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:11:54.931513 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:54.931486 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9d684e2-416f-4a45-b35b-e7666e064fd4-maas-api-xks-ca" pod "a9d684e2-416f-4a45-b35b-e7666e064fd4" (UID: "a9d684e2-416f-4a45-b35b-e7666e064fd4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:11:54.931600 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:54.931541 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9d684e2-416f-4a45-b35b-e7666e064fd4-kube-api-access-ndjcx" pod "a9d684e2-416f-4a45-b35b-e7666e064fd4" (UID: "a9d684e2-416f-4a45-b35b-e7666e064fd4"). InnerVolumeSpecName "kube-api-access-ndjcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:11:55.031454 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:55.031425 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndjcx\" (UniqueName: \"kubernetes.io/projected/a9d684e2-416f-4a45-b35b-e7666e064fd4-kube-api-access-ndjcx\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:55.031454 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:55.031453 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9d684e2-416f-4a45-b35b-e7666e064fd4-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:55.031454 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:55.031463 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9d684e2-416f-4a45-b35b-e7666e064fd4-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:55.684530 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:55.684502 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-115a78-599ch" Oct 30 20:11:55.684691 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:55.684527 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-115a78-599ch" event={"ID":"a9d684e2-416f-4a45-b35b-e7666e064fd4","Type":"ContainerDied","Data":"342559faef7315d22827bb1b6e90a67ca7f453d8c982fe94a74c3b6cc1e7cd51"} Oct 30 20:11:55.684691 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:55.684581 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="342559faef7315d22827bb1b6e90a67ca7f453d8c982fe94a74c3b6cc1e7cd51" Oct 30 20:11:57.279058 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:57.279028 4292 scope.go:122] "RemoveContainer" containerID="dcb4f3ec7860cb84f9a5c4c742c0908e7c63d2a03810b408a4a61ef97da3d59e" Oct 30 20:11:57.287861 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:57.287842 4292 scope.go:122] "RemoveContainer" containerID="fe3a091c7b1afd4b18407013504023d6bafa4269f6548026604a0c6122197686" Oct 30 20:11:58.856009 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:58.855975 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-389c4e/praxis-route-389c4e-kserve-7788b879cc-585s7" Oct 30 20:11:58.868672 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:58.868643 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-389c4e/praxis-route-389c4e-kserve-7788b879cc-585s7" Oct 30 20:11:59.718766 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:11:59.718735 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-389c4e/praxis-route-389c4e-kserve-7788b879cc-585s7" Oct 30 20:12:13.220399 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:12:13.220297 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:12:27.517692 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:12:27.517652 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-84ad63-dvdq8"] Oct 30 20:12:27.569712 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:12:27.569678 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-84ad63-dvdq8"] Oct 30 20:12:28.583192 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:12:28.583158 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26d6a553-565d-4095-b600-ea96b0bb46bd" path="/var/lib/kubelet/pods/26d6a553-565d-4095-b600-ea96b0bb46bd/volumes" Oct 30 20:12:37.413558 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:12:37.413523 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaa6d2-74d8499dd4-wfpmb"] Oct 30 20:12:37.414119 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:12:37.413737 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaa6d2-74d8499dd4-wfpmb" podUID="510eee0c-6b42-4e13-83c3-5d91a98eb7aa" containerName="maas-api" containerID="cri-o://f7aad4990e9baf0b6698cbcce55af25c2aff110536194d28891aa633caa12827" gracePeriod=30 Oct 30 20:12:37.740988 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:12:37.740962 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaa6d2-74d8499dd4-wfpmb" Oct 30 20:12:37.816147 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:12:37.816114 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/510eee0c-6b42-4e13-83c3-5d91a98eb7aa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/510eee0c-6b42-4e13-83c3-5d91a98eb7aa-maas-api-tls\") pod \"510eee0c-6b42-4e13-83c3-5d91a98eb7aa\" (UID: \"510eee0c-6b42-4e13-83c3-5d91a98eb7aa\") " Oct 30 20:12:37.816147 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:12:37.816150 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/510eee0c-6b42-4e13-83c3-5d91a98eb7aa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/510eee0c-6b42-4e13-83c3-5d91a98eb7aa-metrics-certs\") pod \"510eee0c-6b42-4e13-83c3-5d91a98eb7aa\" (UID: \"510eee0c-6b42-4e13-83c3-5d91a98eb7aa\") " Oct 30 20:12:37.816463 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:12:37.816182 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/510eee0c-6b42-4e13-83c3-5d91a98eb7aa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/510eee0c-6b42-4e13-83c3-5d91a98eb7aa-openshift-service-ca\") pod \"510eee0c-6b42-4e13-83c3-5d91a98eb7aa\" (UID: \"510eee0c-6b42-4e13-83c3-5d91a98eb7aa\") " Oct 30 20:12:37.816463 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:12:37.816254 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/510eee0c-6b42-4e13-83c3-5d91a98eb7aa-kube-api-access-nvfq6\" (UniqueName: \"kubernetes.io/projected/510eee0c-6b42-4e13-83c3-5d91a98eb7aa-kube-api-access-nvfq6\") pod \"510eee0c-6b42-4e13-83c3-5d91a98eb7aa\" (UID: \"510eee0c-6b42-4e13-83c3-5d91a98eb7aa\") " Oct 30 20:12:37.816709 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:12:37.816662 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/510eee0c-6b42-4e13-83c3-5d91a98eb7aa-openshift-service-ca" pod "510eee0c-6b42-4e13-83c3-5d91a98eb7aa" (UID: "510eee0c-6b42-4e13-83c3-5d91a98eb7aa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:12:37.819016 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:12:37.818986 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/510eee0c-6b42-4e13-83c3-5d91a98eb7aa-maas-api-tls" pod "510eee0c-6b42-4e13-83c3-5d91a98eb7aa" (UID: "510eee0c-6b42-4e13-83c3-5d91a98eb7aa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:12:37.819016 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:12:37.819003 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/510eee0c-6b42-4e13-83c3-5d91a98eb7aa-metrics-certs" pod "510eee0c-6b42-4e13-83c3-5d91a98eb7aa" (UID: "510eee0c-6b42-4e13-83c3-5d91a98eb7aa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:12:37.819470 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:12:37.819447 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/510eee0c-6b42-4e13-83c3-5d91a98eb7aa-kube-api-access-nvfq6" pod "510eee0c-6b42-4e13-83c3-5d91a98eb7aa" (UID: "510eee0c-6b42-4e13-83c3-5d91a98eb7aa"). InnerVolumeSpecName "kube-api-access-nvfq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:12:37.917607 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:12:37.917570 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/510eee0c-6b42-4e13-83c3-5d91a98eb7aa-maas-api-tls\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:37.917607 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:12:37.917605 4292 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/510eee0c-6b42-4e13-83c3-5d91a98eb7aa-metrics-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:37.917607 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:12:37.917616 4292 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/510eee0c-6b42-4e13-83c3-5d91a98eb7aa-openshift-service-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:37.917839 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:12:37.917628 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvfq6\" (UniqueName: \"kubernetes.io/projected/510eee0c-6b42-4e13-83c3-5d91a98eb7aa-kube-api-access-nvfq6\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:37.945823 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:12:37.945792 4292 generic.go:356] "Generic (PLEG): container finished" podID="510eee0c-6b42-4e13-83c3-5d91a98eb7aa" containerID="f7aad4990e9baf0b6698cbcce55af25c2aff110536194d28891aa633caa12827" exitCode=0 Oct 30 20:12:37.946010 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:12:37.945853 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaa6d2-74d8499dd4-wfpmb" Oct 30 20:12:37.946010 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:12:37.945858 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaa6d2-74d8499dd4-wfpmb" event={"ID":"510eee0c-6b42-4e13-83c3-5d91a98eb7aa","Type":"ContainerDied","Data":"f7aad4990e9baf0b6698cbcce55af25c2aff110536194d28891aa633caa12827"} Oct 30 20:12:37.946010 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:12:37.945910 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaa6d2-74d8499dd4-wfpmb" event={"ID":"510eee0c-6b42-4e13-83c3-5d91a98eb7aa","Type":"ContainerDied","Data":"bab5966bcc559344598cee19565e0838a6534dfeacf1a7974149288c3e6dfd35"} Oct 30 20:12:37.946010 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:12:37.945945 4292 scope.go:122] "RemoveContainer" containerID="f7aad4990e9baf0b6698cbcce55af25c2aff110536194d28891aa633caa12827" Oct 30 20:12:37.957336 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:12:37.957315 4292 scope.go:122] "RemoveContainer" containerID="f7aad4990e9baf0b6698cbcce55af25c2aff110536194d28891aa633caa12827" Oct 30 20:12:37.957657 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:12:37.957637 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7aad4990e9baf0b6698cbcce55af25c2aff110536194d28891aa633caa12827\": container with ID starting with f7aad4990e9baf0b6698cbcce55af25c2aff110536194d28891aa633caa12827 not found: ID does not exist" containerID="f7aad4990e9baf0b6698cbcce55af25c2aff110536194d28891aa633caa12827" Oct 30 20:12:37.957715 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:12:37.957665 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7aad4990e9baf0b6698cbcce55af25c2aff110536194d28891aa633caa12827"} err="failed to get container status \"f7aad4990e9baf0b6698cbcce55af25c2aff110536194d28891aa633caa12827\": rpc error: code = NotFound desc = could not find container \"f7aad4990e9baf0b6698cbcce55af25c2aff110536194d28891aa633caa12827\": container with ID starting with f7aad4990e9baf0b6698cbcce55af25c2aff110536194d28891aa633caa12827 not found: ID does not exist" Oct 30 20:12:38.115210 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:12:38.115177 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaa6d2-74d8499dd4-wfpmb"] Oct 30 20:12:38.119379 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:12:38.119325 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-eaa6d2-74d8499dd4-wfpmb"] Oct 30 20:12:38.582679 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:12:38.582648 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="510eee0c-6b42-4e13-83c3-5d91a98eb7aa" path="/var/lib/kubelet/pods/510eee0c-6b42-4e13-83c3-5d91a98eb7aa/volumes" Oct 30 20:12:39.940558 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:12:39.940529 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eaa6d2-dlpwt"] Oct 30 20:12:39.991444 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:12:39.991414 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eaa6d2-dlpwt"] Oct 30 20:12:40.582708 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:12:40.582681 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8905c19-f981-4501-8fc1-0bfb9b5520ef" path="/var/lib/kubelet/pods/f8905c19-f981-4501-8fc1-0bfb9b5520ef/volumes" Oct 30 20:12:43.221273 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:12:43.221240 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:12:57.050750 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:12:57.050719 4292 generic.go:356] "Generic (PLEG): container finished" podID="c6d51da1-a6c7-4060-8da8-74b8b56e949d" containerID="7306e784265f78402f6c9f9a644106ec8cc339a737ad42bf01206674024e8ca2" exitCode=0 Oct 30 20:12:57.051179 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:12:57.050796 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5" event={"ID":"c6d51da1-a6c7-4060-8da8-74b8b56e949d","Type":"ContainerDied","Data":"7306e784265f78402f6c9f9a644106ec8cc339a737ad42bf01206674024e8ca2"} Oct 30 20:12:57.051224 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:12:57.051198 4292 scope.go:122] "RemoveContainer" containerID="7306e784265f78402f6c9f9a644106ec8cc339a737ad42bf01206674024e8ca2" Oct 30 20:12:57.052782 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:12:57.052756 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:12:57.052852 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:12:57.052826 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:12:57.052892 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:12:57.052860 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:12:58.078431 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:12:58.078393 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5" event={"ID":"c6d51da1-a6c7-4060-8da8-74b8b56e949d","Type":"ContainerStarted","Data":"84b12bffa6d75a2e202ac3503c8a6b58f309b6e7334c3e86114db9acde76dc6d"} Oct 30 20:12:58.603013 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:12:58.602984 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5" Oct 30 20:12:58.603284 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:12:58.603261 4292 patch_prober.go:29] interesting pod/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Oct 30 20:12:58.603385 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:12:58.603305 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5" podUID="c6d51da1-a6c7-4060-8da8-74b8b56e949d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Oct 30 20:12:59.603798 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:12:59.603763 4292 patch_prober.go:29] interesting pod/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Oct 30 20:12:59.604238 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:12:59.603816 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5" podUID="c6d51da1-a6c7-4060-8da8-74b8b56e949d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Oct 30 20:13:00.604046 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:00.604015 4292 patch_prober.go:29] interesting pod/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Oct 30 20:13:00.604579 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:00.604069 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5" podUID="c6d51da1-a6c7-4060-8da8-74b8b56e949d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Oct 30 20:13:01.603371 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:01.603318 4292 patch_prober.go:29] interesting pod/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Oct 30 20:13:01.603546 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:01.603388 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5" podUID="c6d51da1-a6c7-4060-8da8-74b8b56e949d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Oct 30 20:13:02.604018 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:02.603983 4292 patch_prober.go:29] interesting pod/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Oct 30 20:13:02.604465 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:02.604057 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5" podUID="c6d51da1-a6c7-4060-8da8-74b8b56e949d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Oct 30 20:13:03.607443 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:03.607410 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5" Oct 30 20:13:03.607882 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:03.607711 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5" Oct 30 20:13:03.608424 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:03.608406 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5" Oct 30 20:13:10.007619 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:10.007587 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-115a78-599ch"] Oct 30 20:13:10.058204 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:10.058173 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-115a78-599ch"] Oct 30 20:13:10.582252 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:10.582222 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9d684e2-416f-4a45-b35b-e7666e064fd4" path="/var/lib/kubelet/pods/a9d684e2-416f-4a45-b35b-e7666e064fd4/volumes" Oct 30 20:13:13.220706 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:13:13.220658 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:13:24.635836 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:24.635801 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a63e7c65-hl8c5"] Oct 30 20:13:24.636243 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:24.636227 4292 cpu_manager.go:414] "containerMap: removing container" podUID="510eee0c-6b42-4e13-83c3-5d91a98eb7aa" containerName="maas-api" Oct 30 20:13:24.636243 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:24.636237 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="510eee0c-6b42-4e13-83c3-5d91a98eb7aa" containerName="maas-api" Oct 30 20:13:24.636333 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:24.636255 4292 cpu_manager.go:414] "containerMap: removing container" podUID="a9d684e2-416f-4a45-b35b-e7666e064fd4" containerName="revoke-keys" Oct 30 20:13:24.636333 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:24.636261 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9d684e2-416f-4a45-b35b-e7666e064fd4" containerName="revoke-keys" Oct 30 20:13:24.636443 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:24.636336 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9d684e2-416f-4a45-b35b-e7666e064fd4" containerName="revoke-keys" Oct 30 20:13:24.636443 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:24.636362 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="510eee0c-6b42-4e13-83c3-5d91a98eb7aa" containerName="maas-api" Oct 30 20:13:24.638695 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:24.638678 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63e7c65-hl8c5" Oct 30 20:13:24.695291 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:24.695257 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a63e7c65-hl8c5"] Oct 30 20:13:24.742376 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:24.742336 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:13:24.742556 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:24.742513 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\"" Oct 30 20:13:24.797346 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:24.797315 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0587dd2-1d22-4a53-9b86-79cb6bcbe1a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a63e7c65-hl8c5\" (UID: \"a0587dd2-1d22-4a53-9b86-79cb6bcbe1a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63e7c65-hl8c5" Oct 30 20:13:24.797516 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:24.797483 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bftc\" (UniqueName: \"kubernetes.io/projected/a0587dd2-1d22-4a53-9b86-79cb6bcbe1a2-kube-api-access-7bftc\") pod \"maas-api-revoke-sub-a63e7c65-hl8c5\" (UID: \"a0587dd2-1d22-4a53-9b86-79cb6bcbe1a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63e7c65-hl8c5" Oct 30 20:13:24.797516 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:24.797509 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0587dd2-1d22-4a53-9b86-79cb6bcbe1a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a63e7c65-hl8c5\" (UID: \"a0587dd2-1d22-4a53-9b86-79cb6bcbe1a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63e7c65-hl8c5" Oct 30 20:13:24.898802 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:24.898723 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bftc\" (UniqueName: \"kubernetes.io/projected/a0587dd2-1d22-4a53-9b86-79cb6bcbe1a2-kube-api-access-7bftc\") pod \"maas-api-revoke-sub-a63e7c65-hl8c5\" (UID: \"a0587dd2-1d22-4a53-9b86-79cb6bcbe1a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63e7c65-hl8c5" Oct 30 20:13:24.898802 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:24.898762 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0587dd2-1d22-4a53-9b86-79cb6bcbe1a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a63e7c65-hl8c5\" (UID: \"a0587dd2-1d22-4a53-9b86-79cb6bcbe1a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63e7c65-hl8c5" Oct 30 20:13:24.898997 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:24.898876 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0587dd2-1d22-4a53-9b86-79cb6bcbe1a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a63e7c65-hl8c5\" (UID: \"a0587dd2-1d22-4a53-9b86-79cb6bcbe1a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63e7c65-hl8c5" Oct 30 20:13:24.899602 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:24.899574 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0587dd2-1d22-4a53-9b86-79cb6bcbe1a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a63e7c65-hl8c5\" (UID: \"a0587dd2-1d22-4a53-9b86-79cb6bcbe1a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63e7c65-hl8c5" Oct 30 20:13:24.901060 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:24.901042 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0587dd2-1d22-4a53-9b86-79cb6bcbe1a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a63e7c65-hl8c5\" (UID: \"a0587dd2-1d22-4a53-9b86-79cb6bcbe1a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63e7c65-hl8c5" Oct 30 20:13:24.961444 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:24.961414 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bftc\" (UniqueName: \"kubernetes.io/projected/a0587dd2-1d22-4a53-9b86-79cb6bcbe1a2-kube-api-access-7bftc\") pod \"maas-api-revoke-sub-a63e7c65-hl8c5\" (UID: \"a0587dd2-1d22-4a53-9b86-79cb6bcbe1a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63e7c65-hl8c5" Oct 30 20:13:25.250511 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:25.250426 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63e7c65-hl8c5" Oct 30 20:13:25.625774 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:25.625744 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a63e7c65-hl8c5"] Oct 30 20:13:25.627805 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:13:25.627773 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0587dd2_1d22_4a53_9b86_79cb6bcbe1a2.slice/crio-f001f3c4367e67c838751bc435047e8dc54373c917395ea6d739dea68df62b5e WatchSource:0}: Error finding container f001f3c4367e67c838751bc435047e8dc54373c917395ea6d739dea68df62b5e: Status 404 returned error can't find the container with id f001f3c4367e67c838751bc435047e8dc54373c917395ea6d739dea68df62b5e Oct 30 20:13:26.260153 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:26.260065 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63e7c65-hl8c5" event={"ID":"a0587dd2-1d22-4a53-9b86-79cb6bcbe1a2","Type":"ContainerStarted","Data":"b01c1773fcc5413240d8767fe57b5d3830087f671959521ef594713dcab92a8d"} Oct 30 20:13:26.260153 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:26.260128 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63e7c65-hl8c5" event={"ID":"a0587dd2-1d22-4a53-9b86-79cb6bcbe1a2","Type":"ContainerStarted","Data":"f001f3c4367e67c838751bc435047e8dc54373c917395ea6d739dea68df62b5e"} Oct 30 20:13:26.381142 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:26.381090 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63e7c65-hl8c5" podStartSLOduration=2.381077587 podStartE2EDuration="2.381077587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:13:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:13:26.378283268 +0000 UTC m=+2909.896598726" watchObservedRunningTime="2026-10-30 20:13:26.381077587 +0000 UTC m=+2909.899393044" Oct 30 20:13:27.267320 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:27.267288 4292 generic.go:356] "Generic (PLEG): container finished" podID="a0587dd2-1d22-4a53-9b86-79cb6bcbe1a2" containerID="b01c1773fcc5413240d8767fe57b5d3830087f671959521ef594713dcab92a8d" exitCode=0 Oct 30 20:13:27.267747 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:27.267335 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63e7c65-hl8c5" event={"ID":"a0587dd2-1d22-4a53-9b86-79cb6bcbe1a2","Type":"ContainerDied","Data":"b01c1773fcc5413240d8767fe57b5d3830087f671959521ef594713dcab92a8d"} Oct 30 20:13:28.411874 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:28.411848 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63e7c65-hl8c5" Oct 30 20:13:28.541492 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:28.541416 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0587dd2-1d22-4a53-9b86-79cb6bcbe1a2-kube-api-access-7bftc\" (UniqueName: \"kubernetes.io/projected/a0587dd2-1d22-4a53-9b86-79cb6bcbe1a2-kube-api-access-7bftc\") pod \"a0587dd2-1d22-4a53-9b86-79cb6bcbe1a2\" (UID: \"a0587dd2-1d22-4a53-9b86-79cb6bcbe1a2\") " Oct 30 20:13:28.541654 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:28.541514 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0587dd2-1d22-4a53-9b86-79cb6bcbe1a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0587dd2-1d22-4a53-9b86-79cb6bcbe1a2-maas-api-xks-ca\") pod \"a0587dd2-1d22-4a53-9b86-79cb6bcbe1a2\" (UID: \"a0587dd2-1d22-4a53-9b86-79cb6bcbe1a2\") " Oct 30 20:13:28.541793 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:28.541686 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0587dd2-1d22-4a53-9b86-79cb6bcbe1a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0587dd2-1d22-4a53-9b86-79cb6bcbe1a2-maas-api-openshift-ca\") pod \"a0587dd2-1d22-4a53-9b86-79cb6bcbe1a2\" (UID: \"a0587dd2-1d22-4a53-9b86-79cb6bcbe1a2\") " Oct 30 20:13:28.542079 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:28.542050 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0587dd2-1d22-4a53-9b86-79cb6bcbe1a2-maas-api-openshift-ca" pod "a0587dd2-1d22-4a53-9b86-79cb6bcbe1a2" (UID: "a0587dd2-1d22-4a53-9b86-79cb6bcbe1a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:13:28.543636 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:28.543580 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0587dd2-1d22-4a53-9b86-79cb6bcbe1a2-maas-api-xks-ca" pod "a0587dd2-1d22-4a53-9b86-79cb6bcbe1a2" (UID: "a0587dd2-1d22-4a53-9b86-79cb6bcbe1a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:13:28.544065 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:28.544043 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0587dd2-1d22-4a53-9b86-79cb6bcbe1a2-kube-api-access-7bftc" pod "a0587dd2-1d22-4a53-9b86-79cb6bcbe1a2" (UID: "a0587dd2-1d22-4a53-9b86-79cb6bcbe1a2"). InnerVolumeSpecName "kube-api-access-7bftc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:13:28.643238 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:28.643202 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0587dd2-1d22-4a53-9b86-79cb6bcbe1a2-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:28.643238 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:28.643236 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bftc\" (UniqueName: \"kubernetes.io/projected/a0587dd2-1d22-4a53-9b86-79cb6bcbe1a2-kube-api-access-7bftc\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:28.643503 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:28.643251 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0587dd2-1d22-4a53-9b86-79cb6bcbe1a2-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:29.278595 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:29.278564 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63e7c65-hl8c5" event={"ID":"a0587dd2-1d22-4a53-9b86-79cb6bcbe1a2","Type":"ContainerDied","Data":"f001f3c4367e67c838751bc435047e8dc54373c917395ea6d739dea68df62b5e"} Oct 30 20:13:29.278767 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:29.278611 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f001f3c4367e67c838751bc435047e8dc54373c917395ea6d739dea68df62b5e" Oct 30 20:13:29.278767 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:29.278575 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a63e7c65-hl8c5" Oct 30 20:13:29.905770 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:29.905734 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c60685-sszrz"] Oct 30 20:13:29.906196 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:29.906182 4292 cpu_manager.go:414] "containerMap: removing container" podUID="a0587dd2-1d22-4a53-9b86-79cb6bcbe1a2" containerName="revoke-keys" Oct 30 20:13:29.906251 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:29.906198 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0587dd2-1d22-4a53-9b86-79cb6bcbe1a2" containerName="revoke-keys" Oct 30 20:13:29.906297 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:29.906276 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0587dd2-1d22-4a53-9b86-79cb6bcbe1a2" containerName="revoke-keys" Oct 30 20:13:29.908416 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:29.908386 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c60685-sszrz" Oct 30 20:13:29.964332 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:29.964302 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c60685-sszrz"] Oct 30 20:13:30.012363 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:30.012330 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:13:30.012526 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:30.012346 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\"" Oct 30 20:13:30.060170 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:30.060139 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cd75241-e9f7-406f-9984-b4f319df9153-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c60685-sszrz\" (UID: \"3cd75241-e9f7-406f-9984-b4f319df9153\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c60685-sszrz" Oct 30 20:13:30.060317 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:30.060250 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8phf\" (UniqueName: \"kubernetes.io/projected/3cd75241-e9f7-406f-9984-b4f319df9153-kube-api-access-v8phf\") pod \"maas-api-revoke-keys-e2e-ipp-b-c60685-sszrz\" (UID: \"3cd75241-e9f7-406f-9984-b4f319df9153\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c60685-sszrz" Oct 30 20:13:30.060398 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:30.060314 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cd75241-e9f7-406f-9984-b4f319df9153-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c60685-sszrz\" (UID: \"3cd75241-e9f7-406f-9984-b4f319df9153\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c60685-sszrz" Oct 30 20:13:30.161399 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:30.161315 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cd75241-e9f7-406f-9984-b4f319df9153-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c60685-sszrz\" (UID: \"3cd75241-e9f7-406f-9984-b4f319df9153\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c60685-sszrz" Oct 30 20:13:30.161541 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:30.161437 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cd75241-e9f7-406f-9984-b4f319df9153-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c60685-sszrz\" (UID: \"3cd75241-e9f7-406f-9984-b4f319df9153\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c60685-sszrz" Oct 30 20:13:30.161541 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:30.161528 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8phf\" (UniqueName: \"kubernetes.io/projected/3cd75241-e9f7-406f-9984-b4f319df9153-kube-api-access-v8phf\") pod \"maas-api-revoke-keys-e2e-ipp-b-c60685-sszrz\" (UID: \"3cd75241-e9f7-406f-9984-b4f319df9153\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c60685-sszrz" Oct 30 20:13:30.162106 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:30.162082 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cd75241-e9f7-406f-9984-b4f319df9153-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c60685-sszrz\" (UID: \"3cd75241-e9f7-406f-9984-b4f319df9153\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c60685-sszrz" Oct 30 20:13:30.163646 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:30.163622 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cd75241-e9f7-406f-9984-b4f319df9153-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c60685-sszrz\" (UID: \"3cd75241-e9f7-406f-9984-b4f319df9153\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c60685-sszrz" Oct 30 20:13:30.223874 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:30.223841 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8phf\" (UniqueName: \"kubernetes.io/projected/3cd75241-e9f7-406f-9984-b4f319df9153-kube-api-access-v8phf\") pod \"maas-api-revoke-keys-e2e-ipp-b-c60685-sszrz\" (UID: \"3cd75241-e9f7-406f-9984-b4f319df9153\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c60685-sszrz" Oct 30 20:13:30.520242 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:30.520217 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c60685-sszrz" Oct 30 20:13:30.902552 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:30.902528 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c60685-sszrz"] Oct 30 20:13:30.903845 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:13:30.903818 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cd75241_e9f7_406f_9984_b4f319df9153.slice/crio-e4c325218ca3e7cbfb575dca549c640b5f6e377a95ccb93ff497931fa24af122 WatchSource:0}: Error finding container e4c325218ca3e7cbfb575dca549c640b5f6e377a95ccb93ff497931fa24af122: Status 404 returned error can't find the container with id e4c325218ca3e7cbfb575dca549c640b5f6e377a95ccb93ff497931fa24af122 Oct 30 20:13:31.292016 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:31.291981 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c60685-sszrz" event={"ID":"3cd75241-e9f7-406f-9984-b4f319df9153","Type":"ContainerStarted","Data":"cf158f2908fbdc7271ff07c1e8b6635feb8140eabbf8b1d52641881a46389cb1"} Oct 30 20:13:31.292475 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:31.292029 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c60685-sszrz" event={"ID":"3cd75241-e9f7-406f-9984-b4f319df9153","Type":"ContainerStarted","Data":"e4c325218ca3e7cbfb575dca549c640b5f6e377a95ccb93ff497931fa24af122"} Oct 30 20:13:31.411032 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:31.410932 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c60685-sszrz" podStartSLOduration=2.410918236 podStartE2EDuration="2.410918236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:13:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:13:31.40982611 +0000 UTC m=+2914.928141567" watchObservedRunningTime="2026-10-30 20:13:31.410918236 +0000 UTC m=+2914.929233693" Oct 30 20:13:32.298872 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:32.298841 4292 generic.go:356] "Generic (PLEG): container finished" podID="3cd75241-e9f7-406f-9984-b4f319df9153" containerID="cf158f2908fbdc7271ff07c1e8b6635feb8140eabbf8b1d52641881a46389cb1" exitCode=0 Oct 30 20:13:32.299282 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:32.298930 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c60685-sszrz" event={"ID":"3cd75241-e9f7-406f-9984-b4f319df9153","Type":"ContainerDied","Data":"cf158f2908fbdc7271ff07c1e8b6635feb8140eabbf8b1d52641881a46389cb1"} Oct 30 20:13:33.434156 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:33.434131 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c60685-sszrz" Oct 30 20:13:33.596339 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:33.596260 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3cd75241-e9f7-406f-9984-b4f319df9153-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cd75241-e9f7-406f-9984-b4f319df9153-maas-api-xks-ca\") pod \"3cd75241-e9f7-406f-9984-b4f319df9153\" (UID: \"3cd75241-e9f7-406f-9984-b4f319df9153\") " Oct 30 20:13:33.596339 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:33.596326 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cd75241-e9f7-406f-9984-b4f319df9153-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cd75241-e9f7-406f-9984-b4f319df9153-maas-api-openshift-ca\") pod \"3cd75241-e9f7-406f-9984-b4f319df9153\" (UID: \"3cd75241-e9f7-406f-9984-b4f319df9153\") " Oct 30 20:13:33.596542 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:33.596441 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cd75241-e9f7-406f-9984-b4f319df9153-kube-api-access-v8phf\" (UniqueName: \"kubernetes.io/projected/3cd75241-e9f7-406f-9984-b4f319df9153-kube-api-access-v8phf\") pod \"3cd75241-e9f7-406f-9984-b4f319df9153\" (UID: \"3cd75241-e9f7-406f-9984-b4f319df9153\") " Oct 30 20:13:33.596704 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:33.596671 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cd75241-e9f7-406f-9984-b4f319df9153-maas-api-openshift-ca" pod "3cd75241-e9f7-406f-9984-b4f319df9153" (UID: "3cd75241-e9f7-406f-9984-b4f319df9153"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:13:33.596867 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:33.596852 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cd75241-e9f7-406f-9984-b4f319df9153-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:33.598324 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:33.598305 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cd75241-e9f7-406f-9984-b4f319df9153-maas-api-xks-ca" pod "3cd75241-e9f7-406f-9984-b4f319df9153" (UID: "3cd75241-e9f7-406f-9984-b4f319df9153"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:13:33.598452 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:33.598416 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cd75241-e9f7-406f-9984-b4f319df9153-kube-api-access-v8phf" pod "3cd75241-e9f7-406f-9984-b4f319df9153" (UID: "3cd75241-e9f7-406f-9984-b4f319df9153"). InnerVolumeSpecName "kube-api-access-v8phf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:13:33.697900 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:33.697873 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8phf\" (UniqueName: \"kubernetes.io/projected/3cd75241-e9f7-406f-9984-b4f319df9153-kube-api-access-v8phf\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:33.697900 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:33.697898 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cd75241-e9f7-406f-9984-b4f319df9153-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:34.310723 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:34.310681 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c60685-sszrz" event={"ID":"3cd75241-e9f7-406f-9984-b4f319df9153","Type":"ContainerDied","Data":"e4c325218ca3e7cbfb575dca549c640b5f6e377a95ccb93ff497931fa24af122"} Oct 30 20:13:34.310924 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:34.310737 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4c325218ca3e7cbfb575dca549c640b5f6e377a95ccb93ff497931fa24af122" Oct 30 20:13:34.310924 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:34.310697 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c60685-sszrz" Oct 30 20:13:43.015387 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:43.015292 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b01d39e0-qk2s8"] Oct 30 20:13:43.015785 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:43.015769 4292 cpu_manager.go:414] "containerMap: removing container" podUID="3cd75241-e9f7-406f-9984-b4f319df9153" containerName="revoke-keys" Oct 30 20:13:43.015835 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:43.015786 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cd75241-e9f7-406f-9984-b4f319df9153" containerName="revoke-keys" Oct 30 20:13:43.015871 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:43.015865 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cd75241-e9f7-406f-9984-b4f319df9153" containerName="revoke-keys" Oct 30 20:13:43.018202 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:43.018182 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b01d39e0-qk2s8" Oct 30 20:13:43.074961 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:43.074930 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b01d39e0-qk2s8"] Oct 30 20:13:43.086122 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:43.086095 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04d2de80-0f9a-4a90-88b8-284e001b375c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b01d39e0-qk2s8\" (UID: \"04d2de80-0f9a-4a90-88b8-284e001b375c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b01d39e0-qk2s8" Oct 30 20:13:43.086265 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:43.086182 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04d2de80-0f9a-4a90-88b8-284e001b375c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b01d39e0-qk2s8\" (UID: \"04d2de80-0f9a-4a90-88b8-284e001b375c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b01d39e0-qk2s8" Oct 30 20:13:43.086363 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:43.086328 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58jkz\" (UniqueName: \"kubernetes.io/projected/04d2de80-0f9a-4a90-88b8-284e001b375c-kube-api-access-58jkz\") pod \"maas-api-revoke-sub-b01d39e0-qk2s8\" (UID: \"04d2de80-0f9a-4a90-88b8-284e001b375c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b01d39e0-qk2s8" Oct 30 20:13:43.121773 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:43.121689 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\"" Oct 30 20:13:43.121773 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:43.121753 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:13:43.187791 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:43.187760 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04d2de80-0f9a-4a90-88b8-284e001b375c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b01d39e0-qk2s8\" (UID: \"04d2de80-0f9a-4a90-88b8-284e001b375c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b01d39e0-qk2s8" Oct 30 20:13:43.187930 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:43.187840 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04d2de80-0f9a-4a90-88b8-284e001b375c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b01d39e0-qk2s8\" (UID: \"04d2de80-0f9a-4a90-88b8-284e001b375c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b01d39e0-qk2s8" Oct 30 20:13:43.187981 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:43.187952 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58jkz\" (UniqueName: \"kubernetes.io/projected/04d2de80-0f9a-4a90-88b8-284e001b375c-kube-api-access-58jkz\") pod \"maas-api-revoke-sub-b01d39e0-qk2s8\" (UID: \"04d2de80-0f9a-4a90-88b8-284e001b375c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b01d39e0-qk2s8" Oct 30 20:13:43.188614 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:43.188590 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04d2de80-0f9a-4a90-88b8-284e001b375c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b01d39e0-qk2s8\" (UID: \"04d2de80-0f9a-4a90-88b8-284e001b375c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b01d39e0-qk2s8" Oct 30 20:13:43.190110 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:43.190091 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04d2de80-0f9a-4a90-88b8-284e001b375c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b01d39e0-qk2s8\" (UID: \"04d2de80-0f9a-4a90-88b8-284e001b375c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b01d39e0-qk2s8" Oct 30 20:13:43.220480 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:13:43.220445 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:13:43.250673 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:43.250645 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58jkz\" (UniqueName: \"kubernetes.io/projected/04d2de80-0f9a-4a90-88b8-284e001b375c-kube-api-access-58jkz\") pod \"maas-api-revoke-sub-b01d39e0-qk2s8\" (UID: \"04d2de80-0f9a-4a90-88b8-284e001b375c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b01d39e0-qk2s8" Oct 30 20:13:43.329973 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:43.329939 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b01d39e0-qk2s8" Oct 30 20:13:43.720429 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:43.720405 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b01d39e0-qk2s8"] Oct 30 20:13:43.722125 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:13:43.722099 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04d2de80_0f9a_4a90_88b8_284e001b375c.slice/crio-c718b4cb2a9228135b7cefbf82559826811a0861029ee8be7aace74e1e3a63b0 WatchSource:0}: Error finding container c718b4cb2a9228135b7cefbf82559826811a0861029ee8be7aace74e1e3a63b0: Status 404 returned error can't find the container with id c718b4cb2a9228135b7cefbf82559826811a0861029ee8be7aace74e1e3a63b0 Oct 30 20:13:44.371476 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:44.371443 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b01d39e0-qk2s8" event={"ID":"04d2de80-0f9a-4a90-88b8-284e001b375c","Type":"ContainerStarted","Data":"97d9e7a15d2c091367a4569bef153a2cd9927ac7a426117679946bb400485a25"} Oct 30 20:13:44.371873 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:44.371496 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b01d39e0-qk2s8" event={"ID":"04d2de80-0f9a-4a90-88b8-284e001b375c","Type":"ContainerStarted","Data":"c718b4cb2a9228135b7cefbf82559826811a0861029ee8be7aace74e1e3a63b0"} Oct 30 20:13:44.501343 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:44.500345 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b01d39e0-qk2s8" podStartSLOduration=2.500321334 podStartE2EDuration="2.500321334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:13:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:13:44.496128557 +0000 UTC m=+2928.014444025" watchObservedRunningTime="2026-10-30 20:13:44.500321334 +0000 UTC m=+2928.018636795" Oct 30 20:13:45.377810 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:45.377769 4292 generic.go:356] "Generic (PLEG): container finished" podID="04d2de80-0f9a-4a90-88b8-284e001b375c" containerID="97d9e7a15d2c091367a4569bef153a2cd9927ac7a426117679946bb400485a25" exitCode=0 Oct 30 20:13:45.378206 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:45.377853 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b01d39e0-qk2s8" event={"ID":"04d2de80-0f9a-4a90-88b8-284e001b375c","Type":"ContainerDied","Data":"97d9e7a15d2c091367a4569bef153a2cd9927ac7a426117679946bb400485a25"} Oct 30 20:13:46.513673 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:46.513650 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b01d39e0-qk2s8" Oct 30 20:13:46.640367 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:46.640281 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04d2de80-0f9a-4a90-88b8-284e001b375c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04d2de80-0f9a-4a90-88b8-284e001b375c-maas-api-openshift-ca\") pod \"04d2de80-0f9a-4a90-88b8-284e001b375c\" (UID: \"04d2de80-0f9a-4a90-88b8-284e001b375c\") " Oct 30 20:13:46.640367 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:46.640343 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04d2de80-0f9a-4a90-88b8-284e001b375c-kube-api-access-58jkz\" (UniqueName: \"kubernetes.io/projected/04d2de80-0f9a-4a90-88b8-284e001b375c-kube-api-access-58jkz\") pod \"04d2de80-0f9a-4a90-88b8-284e001b375c\" (UID: \"04d2de80-0f9a-4a90-88b8-284e001b375c\") " Oct 30 20:13:46.640562 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:46.640459 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04d2de80-0f9a-4a90-88b8-284e001b375c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04d2de80-0f9a-4a90-88b8-284e001b375c-maas-api-xks-ca\") pod \"04d2de80-0f9a-4a90-88b8-284e001b375c\" (UID: \"04d2de80-0f9a-4a90-88b8-284e001b375c\") " Oct 30 20:13:46.640651 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:46.640629 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04d2de80-0f9a-4a90-88b8-284e001b375c-maas-api-openshift-ca" pod "04d2de80-0f9a-4a90-88b8-284e001b375c" (UID: "04d2de80-0f9a-4a90-88b8-284e001b375c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:13:46.641067 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:46.641043 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04d2de80-0f9a-4a90-88b8-284e001b375c-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:46.642612 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:46.642584 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04d2de80-0f9a-4a90-88b8-284e001b375c-maas-api-xks-ca" pod "04d2de80-0f9a-4a90-88b8-284e001b375c" (UID: "04d2de80-0f9a-4a90-88b8-284e001b375c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:13:46.642804 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:46.642786 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04d2de80-0f9a-4a90-88b8-284e001b375c-kube-api-access-58jkz" pod "04d2de80-0f9a-4a90-88b8-284e001b375c" (UID: "04d2de80-0f9a-4a90-88b8-284e001b375c"). InnerVolumeSpecName "kube-api-access-58jkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:13:46.741889 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:46.741857 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58jkz\" (UniqueName: \"kubernetes.io/projected/04d2de80-0f9a-4a90-88b8-284e001b375c-kube-api-access-58jkz\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:46.741889 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:46.741886 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04d2de80-0f9a-4a90-88b8-284e001b375c-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:47.390194 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:47.390150 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b01d39e0-qk2s8" event={"ID":"04d2de80-0f9a-4a90-88b8-284e001b375c","Type":"ContainerDied","Data":"c718b4cb2a9228135b7cefbf82559826811a0861029ee8be7aace74e1e3a63b0"} Oct 30 20:13:47.390194 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:47.390181 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b01d39e0-qk2s8" Oct 30 20:13:47.390436 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:47.390210 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c718b4cb2a9228135b7cefbf82559826811a0861029ee8be7aace74e1e3a63b0" Oct 30 20:13:49.559154 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:49.559121 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e6b14d-mztks"] Oct 30 20:13:49.559620 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:49.559606 4292 cpu_manager.go:414] "containerMap: removing container" podUID="04d2de80-0f9a-4a90-88b8-284e001b375c" containerName="revoke-keys" Oct 30 20:13:49.559666 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:49.559621 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04d2de80-0f9a-4a90-88b8-284e001b375c" containerName="revoke-keys" Oct 30 20:13:49.559724 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:49.559715 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="04d2de80-0f9a-4a90-88b8-284e001b375c" containerName="revoke-keys" Oct 30 20:13:49.561737 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:49.561722 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e6b14d-mztks" Oct 30 20:13:49.622929 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:49.622894 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e6b14d-mztks"] Oct 30 20:13:49.669832 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:49.669786 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:13:49.670036 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:49.670020 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\"" Oct 30 20:13:49.673686 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:49.673662 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9f6c\" (UniqueName: \"kubernetes.io/projected/1ffe6f78-11ec-4efd-aba7-30411430be0d-kube-api-access-k9f6c\") pod \"maas-api-revoke-keys-e2e-praxis-a-e6b14d-mztks\" (UID: \"1ffe6f78-11ec-4efd-aba7-30411430be0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e6b14d-mztks" Oct 30 20:13:49.673780 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:49.673696 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ffe6f78-11ec-4efd-aba7-30411430be0d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-e6b14d-mztks\" (UID: \"1ffe6f78-11ec-4efd-aba7-30411430be0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e6b14d-mztks" Oct 30 20:13:49.673826 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:49.673789 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ffe6f78-11ec-4efd-aba7-30411430be0d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-e6b14d-mztks\" (UID: \"1ffe6f78-11ec-4efd-aba7-30411430be0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e6b14d-mztks" Oct 30 20:13:49.775135 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:49.775097 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9f6c\" (UniqueName: \"kubernetes.io/projected/1ffe6f78-11ec-4efd-aba7-30411430be0d-kube-api-access-k9f6c\") pod \"maas-api-revoke-keys-e2e-praxis-a-e6b14d-mztks\" (UID: \"1ffe6f78-11ec-4efd-aba7-30411430be0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e6b14d-mztks" Oct 30 20:13:49.775313 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:49.775141 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ffe6f78-11ec-4efd-aba7-30411430be0d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-e6b14d-mztks\" (UID: \"1ffe6f78-11ec-4efd-aba7-30411430be0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e6b14d-mztks" Oct 30 20:13:49.775313 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:49.775188 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ffe6f78-11ec-4efd-aba7-30411430be0d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-e6b14d-mztks\" (UID: \"1ffe6f78-11ec-4efd-aba7-30411430be0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e6b14d-mztks" Oct 30 20:13:49.775868 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:49.775845 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ffe6f78-11ec-4efd-aba7-30411430be0d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-e6b14d-mztks\" (UID: \"1ffe6f78-11ec-4efd-aba7-30411430be0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e6b14d-mztks" Oct 30 20:13:49.777379 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:49.777340 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ffe6f78-11ec-4efd-aba7-30411430be0d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-e6b14d-mztks\" (UID: \"1ffe6f78-11ec-4efd-aba7-30411430be0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e6b14d-mztks" Oct 30 20:13:49.837963 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:49.837895 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9f6c\" (UniqueName: \"kubernetes.io/projected/1ffe6f78-11ec-4efd-aba7-30411430be0d-kube-api-access-k9f6c\") pod \"maas-api-revoke-keys-e2e-praxis-a-e6b14d-mztks\" (UID: \"1ffe6f78-11ec-4efd-aba7-30411430be0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e6b14d-mztks" Oct 30 20:13:49.876359 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:49.876318 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e6b14d-mztks" Oct 30 20:13:50.252954 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:50.252928 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e6b14d-mztks"] Oct 30 20:13:50.255307 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:13:50.255279 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ffe6f78_11ec_4efd_aba7_30411430be0d.slice/crio-c5844f6c7c35f36303f6f3cba23f4050fb1cbc5c6ab43379a3cd79e9ca94bffc WatchSource:0}: Error finding container c5844f6c7c35f36303f6f3cba23f4050fb1cbc5c6ab43379a3cd79e9ca94bffc: Status 404 returned error can't find the container with id c5844f6c7c35f36303f6f3cba23f4050fb1cbc5c6ab43379a3cd79e9ca94bffc Oct 30 20:13:50.411760 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:50.411726 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e6b14d-mztks" event={"ID":"1ffe6f78-11ec-4efd-aba7-30411430be0d","Type":"ContainerStarted","Data":"c5844f6c7c35f36303f6f3cba23f4050fb1cbc5c6ab43379a3cd79e9ca94bffc"} Oct 30 20:13:51.419764 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:51.419679 4292 generic.go:356] "Generic (PLEG): container finished" podID="1ffe6f78-11ec-4efd-aba7-30411430be0d" containerID="542a219f894044b1e8a0537c579dae8457d7edc0808f083d129710da12de7fa5" exitCode=0 Oct 30 20:13:51.420156 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:51.419766 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e6b14d-mztks" event={"ID":"1ffe6f78-11ec-4efd-aba7-30411430be0d","Type":"ContainerDied","Data":"542a219f894044b1e8a0537c579dae8457d7edc0808f083d129710da12de7fa5"} Oct 30 20:13:52.553556 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:52.553531 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e6b14d-mztks" Oct 30 20:13:52.620362 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:52.620333 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ffe6f78-11ec-4efd-aba7-30411430be0d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ffe6f78-11ec-4efd-aba7-30411430be0d-maas-api-openshift-ca\") pod \"1ffe6f78-11ec-4efd-aba7-30411430be0d\" (UID: \"1ffe6f78-11ec-4efd-aba7-30411430be0d\") " Oct 30 20:13:52.620499 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:52.620481 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ffe6f78-11ec-4efd-aba7-30411430be0d-kube-api-access-k9f6c\" (UniqueName: \"kubernetes.io/projected/1ffe6f78-11ec-4efd-aba7-30411430be0d-kube-api-access-k9f6c\") pod \"1ffe6f78-11ec-4efd-aba7-30411430be0d\" (UID: \"1ffe6f78-11ec-4efd-aba7-30411430be0d\") " Oct 30 20:13:52.620568 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:52.620512 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ffe6f78-11ec-4efd-aba7-30411430be0d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ffe6f78-11ec-4efd-aba7-30411430be0d-maas-api-xks-ca\") pod \"1ffe6f78-11ec-4efd-aba7-30411430be0d\" (UID: \"1ffe6f78-11ec-4efd-aba7-30411430be0d\") " Oct 30 20:13:52.620704 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:52.620681 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ffe6f78-11ec-4efd-aba7-30411430be0d-maas-api-openshift-ca" pod "1ffe6f78-11ec-4efd-aba7-30411430be0d" (UID: "1ffe6f78-11ec-4efd-aba7-30411430be0d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:13:52.620839 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:52.620822 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ffe6f78-11ec-4efd-aba7-30411430be0d-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:52.622519 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:52.622497 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ffe6f78-11ec-4efd-aba7-30411430be0d-maas-api-xks-ca" pod "1ffe6f78-11ec-4efd-aba7-30411430be0d" (UID: "1ffe6f78-11ec-4efd-aba7-30411430be0d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:13:52.622613 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:52.622571 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ffe6f78-11ec-4efd-aba7-30411430be0d-kube-api-access-k9f6c" pod "1ffe6f78-11ec-4efd-aba7-30411430be0d" (UID: "1ffe6f78-11ec-4efd-aba7-30411430be0d"). InnerVolumeSpecName "kube-api-access-k9f6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:13:52.721842 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:52.721772 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9f6c\" (UniqueName: \"kubernetes.io/projected/1ffe6f78-11ec-4efd-aba7-30411430be0d-kube-api-access-k9f6c\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:52.721842 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:52.721802 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ffe6f78-11ec-4efd-aba7-30411430be0d-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:53.439568 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:53.439538 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e6b14d-mztks" Oct 30 20:13:53.439758 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:53.439531 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e6b14d-mztks" event={"ID":"1ffe6f78-11ec-4efd-aba7-30411430be0d","Type":"ContainerDied","Data":"c5844f6c7c35f36303f6f3cba23f4050fb1cbc5c6ab43379a3cd79e9ca94bffc"} Oct 30 20:13:53.439758 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:53.439710 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5844f6c7c35f36303f6f3cba23f4050fb1cbc5c6ab43379a3cd79e9ca94bffc" Oct 30 20:13:59.373428 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:59.373390 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ced19c95-p7v8h"] Oct 30 20:13:59.373958 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:59.373907 4292 cpu_manager.go:414] "containerMap: removing container" podUID="1ffe6f78-11ec-4efd-aba7-30411430be0d" containerName="revoke-keys" Oct 30 20:13:59.373958 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:59.373922 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ffe6f78-11ec-4efd-aba7-30411430be0d" containerName="revoke-keys" Oct 30 20:13:59.374040 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:59.373989 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ffe6f78-11ec-4efd-aba7-30411430be0d" containerName="revoke-keys" Oct 30 20:13:59.376992 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:59.376975 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced19c95-p7v8h" Oct 30 20:13:59.431494 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:59.431465 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ced19c95-p7v8h"] Oct 30 20:13:59.479481 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:59.479455 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:13:59.479627 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:59.479501 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\"" Oct 30 20:13:59.504775 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:59.504750 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/901c66b7-8ea1-41be-9c7c-535fe0e475d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ced19c95-p7v8h\" (UID: \"901c66b7-8ea1-41be-9c7c-535fe0e475d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced19c95-p7v8h" Oct 30 20:13:59.504921 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:59.504826 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr7dm\" (UniqueName: \"kubernetes.io/projected/901c66b7-8ea1-41be-9c7c-535fe0e475d9-kube-api-access-tr7dm\") pod \"maas-api-revoke-sub-ced19c95-p7v8h\" (UID: \"901c66b7-8ea1-41be-9c7c-535fe0e475d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced19c95-p7v8h" Oct 30 20:13:59.504964 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:59.504954 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/901c66b7-8ea1-41be-9c7c-535fe0e475d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ced19c95-p7v8h\" (UID: \"901c66b7-8ea1-41be-9c7c-535fe0e475d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced19c95-p7v8h" Oct 30 20:13:59.606011 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:59.605980 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr7dm\" (UniqueName: \"kubernetes.io/projected/901c66b7-8ea1-41be-9c7c-535fe0e475d9-kube-api-access-tr7dm\") pod \"maas-api-revoke-sub-ced19c95-p7v8h\" (UID: \"901c66b7-8ea1-41be-9c7c-535fe0e475d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced19c95-p7v8h" Oct 30 20:13:59.606175 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:59.606082 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/901c66b7-8ea1-41be-9c7c-535fe0e475d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ced19c95-p7v8h\" (UID: \"901c66b7-8ea1-41be-9c7c-535fe0e475d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced19c95-p7v8h" Oct 30 20:13:59.606175 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:59.606134 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/901c66b7-8ea1-41be-9c7c-535fe0e475d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ced19c95-p7v8h\" (UID: \"901c66b7-8ea1-41be-9c7c-535fe0e475d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced19c95-p7v8h" Oct 30 20:13:59.607020 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:59.606991 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/901c66b7-8ea1-41be-9c7c-535fe0e475d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ced19c95-p7v8h\" (UID: \"901c66b7-8ea1-41be-9c7c-535fe0e475d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced19c95-p7v8h" Oct 30 20:13:59.608599 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:59.608575 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/901c66b7-8ea1-41be-9c7c-535fe0e475d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ced19c95-p7v8h\" (UID: \"901c66b7-8ea1-41be-9c7c-535fe0e475d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced19c95-p7v8h" Oct 30 20:13:59.669477 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:59.669422 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr7dm\" (UniqueName: \"kubernetes.io/projected/901c66b7-8ea1-41be-9c7c-535fe0e475d9-kube-api-access-tr7dm\") pod \"maas-api-revoke-sub-ced19c95-p7v8h\" (UID: \"901c66b7-8ea1-41be-9c7c-535fe0e475d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced19c95-p7v8h" Oct 30 20:13:59.690631 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:59.690602 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced19c95-p7v8h" Oct 30 20:13:59.913285 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:59.913255 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c60685-6977c9cbcb-72dxx"] Oct 30 20:13:59.913508 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:13:59.913468 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c60685-6977c9cbcb-72dxx" podUID="458e28f7-8eb9-45b9-a6db-aa40888a3e2e" containerName="maas-api" containerID="cri-o://65a6132f89b4d2544d2a2684527407a9bf579e6e556dc5f82a66d250928083ab" gracePeriod=30 Oct 30 20:14:00.071166 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:00.071138 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ced19c95-p7v8h"] Oct 30 20:14:00.072283 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:14:00.072254 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod901c66b7_8ea1_41be_9c7c_535fe0e475d9.slice/crio-76850d2049681f2103201c2c183bb4a8b4249a80868032ed11183b14d8acfe44 WatchSource:0}: Error finding container 76850d2049681f2103201c2c183bb4a8b4249a80868032ed11183b14d8acfe44: Status 404 returned error can't find the container with id 76850d2049681f2103201c2c183bb4a8b4249a80868032ed11183b14d8acfe44 Oct 30 20:14:00.173748 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:00.173717 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c60685-6977c9cbcb-72dxx" Oct 30 20:14:00.221301 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:00.221230 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/458e28f7-8eb9-45b9-a6db-aa40888a3e2e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/458e28f7-8eb9-45b9-a6db-aa40888a3e2e-openshift-service-ca\") pod \"458e28f7-8eb9-45b9-a6db-aa40888a3e2e\" (UID: \"458e28f7-8eb9-45b9-a6db-aa40888a3e2e\") " Oct 30 20:14:00.221301 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:00.221283 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/458e28f7-8eb9-45b9-a6db-aa40888a3e2e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/458e28f7-8eb9-45b9-a6db-aa40888a3e2e-maas-api-tls\") pod \"458e28f7-8eb9-45b9-a6db-aa40888a3e2e\" (UID: \"458e28f7-8eb9-45b9-a6db-aa40888a3e2e\") " Oct 30 20:14:00.221510 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:00.221376 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/458e28f7-8eb9-45b9-a6db-aa40888a3e2e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/458e28f7-8eb9-45b9-a6db-aa40888a3e2e-metrics-certs\") pod \"458e28f7-8eb9-45b9-a6db-aa40888a3e2e\" (UID: \"458e28f7-8eb9-45b9-a6db-aa40888a3e2e\") " Oct 30 20:14:00.221510 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:00.221458 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/458e28f7-8eb9-45b9-a6db-aa40888a3e2e-kube-api-access-5c5fm\" (UniqueName: \"kubernetes.io/projected/458e28f7-8eb9-45b9-a6db-aa40888a3e2e-kube-api-access-5c5fm\") pod \"458e28f7-8eb9-45b9-a6db-aa40888a3e2e\" (UID: \"458e28f7-8eb9-45b9-a6db-aa40888a3e2e\") " Oct 30 20:14:00.221622 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:00.221591 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/458e28f7-8eb9-45b9-a6db-aa40888a3e2e-openshift-service-ca" pod "458e28f7-8eb9-45b9-a6db-aa40888a3e2e" (UID: "458e28f7-8eb9-45b9-a6db-aa40888a3e2e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:14:00.221878 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:00.221847 4292 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/458e28f7-8eb9-45b9-a6db-aa40888a3e2e-openshift-service-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:00.223517 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:00.223492 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/458e28f7-8eb9-45b9-a6db-aa40888a3e2e-maas-api-tls" pod "458e28f7-8eb9-45b9-a6db-aa40888a3e2e" (UID: "458e28f7-8eb9-45b9-a6db-aa40888a3e2e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:14:00.223644 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:00.223520 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/458e28f7-8eb9-45b9-a6db-aa40888a3e2e-metrics-certs" pod "458e28f7-8eb9-45b9-a6db-aa40888a3e2e" (UID: "458e28f7-8eb9-45b9-a6db-aa40888a3e2e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:14:00.223644 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:00.223563 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/458e28f7-8eb9-45b9-a6db-aa40888a3e2e-kube-api-access-5c5fm" pod "458e28f7-8eb9-45b9-a6db-aa40888a3e2e" (UID: "458e28f7-8eb9-45b9-a6db-aa40888a3e2e"). InnerVolumeSpecName "kube-api-access-5c5fm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:00.325070 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:00.322719 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5c5fm\" (UniqueName: \"kubernetes.io/projected/458e28f7-8eb9-45b9-a6db-aa40888a3e2e-kube-api-access-5c5fm\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:00.325070 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:00.322798 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/458e28f7-8eb9-45b9-a6db-aa40888a3e2e-maas-api-tls\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:00.325070 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:00.322816 4292 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/458e28f7-8eb9-45b9-a6db-aa40888a3e2e-metrics-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:00.499866 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:00.499798 4292 generic.go:356] "Generic (PLEG): container finished" podID="458e28f7-8eb9-45b9-a6db-aa40888a3e2e" containerID="65a6132f89b4d2544d2a2684527407a9bf579e6e556dc5f82a66d250928083ab" exitCode=0 Oct 30 20:14:00.500295 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:00.499864 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c60685-6977c9cbcb-72dxx" Oct 30 20:14:00.500295 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:00.499880 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c60685-6977c9cbcb-72dxx" event={"ID":"458e28f7-8eb9-45b9-a6db-aa40888a3e2e","Type":"ContainerDied","Data":"65a6132f89b4d2544d2a2684527407a9bf579e6e556dc5f82a66d250928083ab"} Oct 30 20:14:00.500295 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:00.499938 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c60685-6977c9cbcb-72dxx" event={"ID":"458e28f7-8eb9-45b9-a6db-aa40888a3e2e","Type":"ContainerDied","Data":"f0cb873127ea8aed10e951a1f0c5b4f4f0433d929592e3fe5941150e307b2e0d"} Oct 30 20:14:00.500295 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:00.499970 4292 scope.go:122] "RemoveContainer" containerID="65a6132f89b4d2544d2a2684527407a9bf579e6e556dc5f82a66d250928083ab" Oct 30 20:14:00.502167 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:00.502143 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced19c95-p7v8h" event={"ID":"901c66b7-8ea1-41be-9c7c-535fe0e475d9","Type":"ContainerStarted","Data":"1b5bed8f03ef22444d2d63c7d65bc27f3f2757b83054419783f56bd9e7352156"} Oct 30 20:14:00.502302 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:00.502197 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced19c95-p7v8h" event={"ID":"901c66b7-8ea1-41be-9c7c-535fe0e475d9","Type":"ContainerStarted","Data":"76850d2049681f2103201c2c183bb4a8b4249a80868032ed11183b14d8acfe44"} Oct 30 20:14:00.512540 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:00.512325 4292 scope.go:122] "RemoveContainer" containerID="65a6132f89b4d2544d2a2684527407a9bf579e6e556dc5f82a66d250928083ab" Oct 30 20:14:00.513154 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:14:00.513124 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65a6132f89b4d2544d2a2684527407a9bf579e6e556dc5f82a66d250928083ab\": container with ID starting with 65a6132f89b4d2544d2a2684527407a9bf579e6e556dc5f82a66d250928083ab not found: ID does not exist" containerID="65a6132f89b4d2544d2a2684527407a9bf579e6e556dc5f82a66d250928083ab" Oct 30 20:14:00.513267 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:00.513185 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65a6132f89b4d2544d2a2684527407a9bf579e6e556dc5f82a66d250928083ab"} err="failed to get container status \"65a6132f89b4d2544d2a2684527407a9bf579e6e556dc5f82a66d250928083ab\": rpc error: code = NotFound desc = could not find container \"65a6132f89b4d2544d2a2684527407a9bf579e6e556dc5f82a66d250928083ab\": container with ID starting with 65a6132f89b4d2544d2a2684527407a9bf579e6e556dc5f82a66d250928083ab not found: ID does not exist" Oct 30 20:14:00.669436 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:00.669410 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c60685-6977c9cbcb-72dxx"] Oct 30 20:14:00.671750 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:00.671728 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c60685-6977c9cbcb-72dxx"] Oct 30 20:14:00.788329 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:00.788276 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced19c95-p7v8h" podStartSLOduration=1.7882607099999999 podStartE2EDuration="1.78826071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:13:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:14:00.787134944 +0000 UTC m=+2944.305450411" watchObservedRunningTime="2026-10-30 20:14:00.78826071 +0000 UTC m=+2944.306576167" Oct 30 20:14:00.920161 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:00.920129 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21aab5d1-2ph6l"] Oct 30 20:14:00.920663 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:00.920646 4292 cpu_manager.go:414] "containerMap: removing container" podUID="458e28f7-8eb9-45b9-a6db-aa40888a3e2e" containerName="maas-api" Oct 30 20:14:00.920745 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:00.920664 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="458e28f7-8eb9-45b9-a6db-aa40888a3e2e" containerName="maas-api" Oct 30 20:14:00.920788 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:00.920766 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="458e28f7-8eb9-45b9-a6db-aa40888a3e2e" containerName="maas-api" Oct 30 20:14:00.925314 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:00.925287 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21aab5d1-2ph6l" Oct 30 20:14:00.935898 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:00.935875 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8cedf1d-e992-4152-9f8f-2ba60cadc3be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21aab5d1-2ph6l\" (UID: \"d8cedf1d-e992-4152-9f8f-2ba60cadc3be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21aab5d1-2ph6l" Oct 30 20:14:00.936012 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:00.935995 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwrc8\" (UniqueName: \"kubernetes.io/projected/d8cedf1d-e992-4152-9f8f-2ba60cadc3be-kube-api-access-mwrc8\") pod \"maas-api-revoke-sub-21aab5d1-2ph6l\" (UID: \"d8cedf1d-e992-4152-9f8f-2ba60cadc3be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21aab5d1-2ph6l" Oct 30 20:14:00.936187 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:00.936167 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8cedf1d-e992-4152-9f8f-2ba60cadc3be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21aab5d1-2ph6l\" (UID: \"d8cedf1d-e992-4152-9f8f-2ba60cadc3be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21aab5d1-2ph6l" Oct 30 20:14:00.979560 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:00.979526 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21aab5d1-2ph6l"] Oct 30 20:14:01.037030 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:01.037007 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8cedf1d-e992-4152-9f8f-2ba60cadc3be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21aab5d1-2ph6l\" (UID: \"d8cedf1d-e992-4152-9f8f-2ba60cadc3be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21aab5d1-2ph6l" Oct 30 20:14:01.037137 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:01.037065 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8cedf1d-e992-4152-9f8f-2ba60cadc3be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21aab5d1-2ph6l\" (UID: \"d8cedf1d-e992-4152-9f8f-2ba60cadc3be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21aab5d1-2ph6l" Oct 30 20:14:01.037137 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:01.037119 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwrc8\" (UniqueName: \"kubernetes.io/projected/d8cedf1d-e992-4152-9f8f-2ba60cadc3be-kube-api-access-mwrc8\") pod \"maas-api-revoke-sub-21aab5d1-2ph6l\" (UID: \"d8cedf1d-e992-4152-9f8f-2ba60cadc3be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21aab5d1-2ph6l" Oct 30 20:14:01.037717 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:01.037694 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8cedf1d-e992-4152-9f8f-2ba60cadc3be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21aab5d1-2ph6l\" (UID: \"d8cedf1d-e992-4152-9f8f-2ba60cadc3be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21aab5d1-2ph6l" Oct 30 20:14:01.039480 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:01.039434 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8cedf1d-e992-4152-9f8f-2ba60cadc3be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21aab5d1-2ph6l\" (UID: \"d8cedf1d-e992-4152-9f8f-2ba60cadc3be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21aab5d1-2ph6l" Oct 30 20:14:01.100461 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:01.100435 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwrc8\" (UniqueName: \"kubernetes.io/projected/d8cedf1d-e992-4152-9f8f-2ba60cadc3be-kube-api-access-mwrc8\") pod \"maas-api-revoke-sub-21aab5d1-2ph6l\" (UID: \"d8cedf1d-e992-4152-9f8f-2ba60cadc3be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21aab5d1-2ph6l" Oct 30 20:14:01.238883 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:01.238855 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21aab5d1-2ph6l" Oct 30 20:14:01.508585 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:01.508552 4292 generic.go:356] "Generic (PLEG): container finished" podID="901c66b7-8ea1-41be-9c7c-535fe0e475d9" containerID="1b5bed8f03ef22444d2d63c7d65bc27f3f2757b83054419783f56bd9e7352156" exitCode=0 Oct 30 20:14:01.509016 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:01.508632 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced19c95-p7v8h" event={"ID":"901c66b7-8ea1-41be-9c7c-535fe0e475d9","Type":"ContainerDied","Data":"1b5bed8f03ef22444d2d63c7d65bc27f3f2757b83054419783f56bd9e7352156"} Oct 30 20:14:01.620053 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:01.620029 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21aab5d1-2ph6l"] Oct 30 20:14:01.621673 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:14:01.621633 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8cedf1d_e992_4152_9f8f_2ba60cadc3be.slice/crio-7f2b02cfa2186986d336c8b5779e5c136fe64092ecf0ca4bcb9b113535258a73 WatchSource:0}: Error finding container 7f2b02cfa2186986d336c8b5779e5c136fe64092ecf0ca4bcb9b113535258a73: Status 404 returned error can't find the container with id 7f2b02cfa2186986d336c8b5779e5c136fe64092ecf0ca4bcb9b113535258a73 Oct 30 20:14:02.074041 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:02.074010 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c60685-sszrz"] Oct 30 20:14:02.076342 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:02.076321 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c60685-sszrz"] Oct 30 20:14:02.517483 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:02.517448 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21aab5d1-2ph6l" event={"ID":"d8cedf1d-e992-4152-9f8f-2ba60cadc3be","Type":"ContainerStarted","Data":"0f9459221d8021252839f2a17d8194bfbf6c17b148e908eacddf30c85d5a93f2"} Oct 30 20:14:02.517892 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:02.517509 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21aab5d1-2ph6l" event={"ID":"d8cedf1d-e992-4152-9f8f-2ba60cadc3be","Type":"ContainerStarted","Data":"7f2b02cfa2186986d336c8b5779e5c136fe64092ecf0ca4bcb9b113535258a73"} Oct 30 20:14:02.646472 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:02.646430 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cd75241-e9f7-406f-9984-b4f319df9153" path="/var/lib/kubelet/pods/3cd75241-e9f7-406f-9984-b4f319df9153/volumes" Oct 30 20:14:02.647087 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:02.647061 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="458e28f7-8eb9-45b9-a6db-aa40888a3e2e" path="/var/lib/kubelet/pods/458e28f7-8eb9-45b9-a6db-aa40888a3e2e/volumes" Oct 30 20:14:02.749955 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:02.749921 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced19c95-p7v8h" Oct 30 20:14:02.779854 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:02.779771 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/901c66b7-8ea1-41be-9c7c-535fe0e475d9-kube-api-access-tr7dm\" (UniqueName: \"kubernetes.io/projected/901c66b7-8ea1-41be-9c7c-535fe0e475d9-kube-api-access-tr7dm\") pod \"901c66b7-8ea1-41be-9c7c-535fe0e475d9\" (UID: \"901c66b7-8ea1-41be-9c7c-535fe0e475d9\") " Oct 30 20:14:02.779854 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:02.779805 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/901c66b7-8ea1-41be-9c7c-535fe0e475d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/901c66b7-8ea1-41be-9c7c-535fe0e475d9-maas-api-openshift-ca\") pod \"901c66b7-8ea1-41be-9c7c-535fe0e475d9\" (UID: \"901c66b7-8ea1-41be-9c7c-535fe0e475d9\") " Oct 30 20:14:02.779854 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:02.779837 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/901c66b7-8ea1-41be-9c7c-535fe0e475d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/901c66b7-8ea1-41be-9c7c-535fe0e475d9-maas-api-xks-ca\") pod \"901c66b7-8ea1-41be-9c7c-535fe0e475d9\" (UID: \"901c66b7-8ea1-41be-9c7c-535fe0e475d9\") " Oct 30 20:14:02.780208 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:02.780182 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/901c66b7-8ea1-41be-9c7c-535fe0e475d9-maas-api-openshift-ca" pod "901c66b7-8ea1-41be-9c7c-535fe0e475d9" (UID: "901c66b7-8ea1-41be-9c7c-535fe0e475d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:14:02.782029 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:02.782005 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/901c66b7-8ea1-41be-9c7c-535fe0e475d9-maas-api-xks-ca" pod "901c66b7-8ea1-41be-9c7c-535fe0e475d9" (UID: "901c66b7-8ea1-41be-9c7c-535fe0e475d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:14:02.782167 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:02.782039 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/901c66b7-8ea1-41be-9c7c-535fe0e475d9-kube-api-access-tr7dm" pod "901c66b7-8ea1-41be-9c7c-535fe0e475d9" (UID: "901c66b7-8ea1-41be-9c7c-535fe0e475d9"). InnerVolumeSpecName "kube-api-access-tr7dm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:02.872375 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:02.871670 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21aab5d1-2ph6l" podStartSLOduration=2.871648371 podStartE2EDuration="2.871648371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:14:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:14:02.646229848 +0000 UTC m=+2946.164545305" watchObservedRunningTime="2026-10-30 20:14:02.871648371 +0000 UTC m=+2946.389963832" Oct 30 20:14:02.881451 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:02.881420 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tr7dm\" (UniqueName: \"kubernetes.io/projected/901c66b7-8ea1-41be-9c7c-535fe0e475d9-kube-api-access-tr7dm\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:02.881451 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:02.881452 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/901c66b7-8ea1-41be-9c7c-535fe0e475d9-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:02.881686 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:02.881469 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/901c66b7-8ea1-41be-9c7c-535fe0e475d9-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:03.525669 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:03.525636 4292 generic.go:356] "Generic (PLEG): container finished" podID="d8cedf1d-e992-4152-9f8f-2ba60cadc3be" containerID="0f9459221d8021252839f2a17d8194bfbf6c17b148e908eacddf30c85d5a93f2" exitCode=0 Oct 30 20:14:03.526106 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:03.525722 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21aab5d1-2ph6l" event={"ID":"d8cedf1d-e992-4152-9f8f-2ba60cadc3be","Type":"ContainerDied","Data":"0f9459221d8021252839f2a17d8194bfbf6c17b148e908eacddf30c85d5a93f2"} Oct 30 20:14:03.527077 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:03.527052 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced19c95-p7v8h" event={"ID":"901c66b7-8ea1-41be-9c7c-535fe0e475d9","Type":"ContainerDied","Data":"76850d2049681f2103201c2c183bb4a8b4249a80868032ed11183b14d8acfe44"} Oct 30 20:14:03.527209 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:03.527092 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76850d2049681f2103201c2c183bb4a8b4249a80868032ed11183b14d8acfe44" Oct 30 20:14:03.527209 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:03.527095 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced19c95-p7v8h" Oct 30 20:14:04.669998 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:04.669975 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21aab5d1-2ph6l" Oct 30 20:14:04.700382 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:04.700341 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8cedf1d-e992-4152-9f8f-2ba60cadc3be-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8cedf1d-e992-4152-9f8f-2ba60cadc3be-maas-api-openshift-ca\") pod \"d8cedf1d-e992-4152-9f8f-2ba60cadc3be\" (UID: \"d8cedf1d-e992-4152-9f8f-2ba60cadc3be\") " Oct 30 20:14:04.700526 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:04.700419 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8cedf1d-e992-4152-9f8f-2ba60cadc3be-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8cedf1d-e992-4152-9f8f-2ba60cadc3be-maas-api-xks-ca\") pod \"d8cedf1d-e992-4152-9f8f-2ba60cadc3be\" (UID: \"d8cedf1d-e992-4152-9f8f-2ba60cadc3be\") " Oct 30 20:14:04.700621 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:04.700607 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8cedf1d-e992-4152-9f8f-2ba60cadc3be-kube-api-access-mwrc8\" (UniqueName: \"kubernetes.io/projected/d8cedf1d-e992-4152-9f8f-2ba60cadc3be-kube-api-access-mwrc8\") pod \"d8cedf1d-e992-4152-9f8f-2ba60cadc3be\" (UID: \"d8cedf1d-e992-4152-9f8f-2ba60cadc3be\") " Oct 30 20:14:04.700681 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:04.700659 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8cedf1d-e992-4152-9f8f-2ba60cadc3be-maas-api-openshift-ca" pod "d8cedf1d-e992-4152-9f8f-2ba60cadc3be" (UID: "d8cedf1d-e992-4152-9f8f-2ba60cadc3be"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:14:04.700939 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:04.700923 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8cedf1d-e992-4152-9f8f-2ba60cadc3be-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:04.702494 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:04.702473 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8cedf1d-e992-4152-9f8f-2ba60cadc3be-maas-api-xks-ca" pod "d8cedf1d-e992-4152-9f8f-2ba60cadc3be" (UID: "d8cedf1d-e992-4152-9f8f-2ba60cadc3be"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:14:04.702579 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:04.702548 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8cedf1d-e992-4152-9f8f-2ba60cadc3be-kube-api-access-mwrc8" pod "d8cedf1d-e992-4152-9f8f-2ba60cadc3be" (UID: "d8cedf1d-e992-4152-9f8f-2ba60cadc3be"). InnerVolumeSpecName "kube-api-access-mwrc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:04.802237 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:04.802158 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwrc8\" (UniqueName: \"kubernetes.io/projected/d8cedf1d-e992-4152-9f8f-2ba60cadc3be-kube-api-access-mwrc8\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:04.802237 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:04.802186 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8cedf1d-e992-4152-9f8f-2ba60cadc3be-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:05.260814 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:05.260738 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-389c4e-966lm"] Oct 30 20:14:05.261183 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:05.261171 4292 cpu_manager.go:414] "containerMap: removing container" podUID="d8cedf1d-e992-4152-9f8f-2ba60cadc3be" containerName="revoke-keys" Oct 30 20:14:05.261234 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:05.261184 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8cedf1d-e992-4152-9f8f-2ba60cadc3be" containerName="revoke-keys" Oct 30 20:14:05.261234 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:05.261204 4292 cpu_manager.go:414] "containerMap: removing container" podUID="901c66b7-8ea1-41be-9c7c-535fe0e475d9" containerName="revoke-keys" Oct 30 20:14:05.261234 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:05.261209 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="901c66b7-8ea1-41be-9c7c-535fe0e475d9" containerName="revoke-keys" Oct 30 20:14:05.261342 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:05.261279 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8cedf1d-e992-4152-9f8f-2ba60cadc3be" containerName="revoke-keys" Oct 30 20:14:05.261342 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:05.261297 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="901c66b7-8ea1-41be-9c7c-535fe0e475d9" containerName="revoke-keys" Oct 30 20:14:05.264684 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:05.264665 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-389c4e-966lm" Oct 30 20:14:05.307635 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:05.307599 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db133a39-3ac3-49de-91e9-6eac2647b59a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-389c4e-966lm\" (UID: \"db133a39-3ac3-49de-91e9-6eac2647b59a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-389c4e-966lm" Oct 30 20:14:05.307808 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:05.307782 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svf6r\" (UniqueName: \"kubernetes.io/projected/db133a39-3ac3-49de-91e9-6eac2647b59a-kube-api-access-svf6r\") pod \"maas-api-revoke-keys-e2e-praxis-a-389c4e-966lm\" (UID: \"db133a39-3ac3-49de-91e9-6eac2647b59a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-389c4e-966lm" Oct 30 20:14:05.307938 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:05.307913 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db133a39-3ac3-49de-91e9-6eac2647b59a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-389c4e-966lm\" (UID: \"db133a39-3ac3-49de-91e9-6eac2647b59a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-389c4e-966lm" Oct 30 20:14:05.330435 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:05.322157 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-389c4e-966lm"] Oct 30 20:14:05.409193 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:05.409159 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svf6r\" (UniqueName: \"kubernetes.io/projected/db133a39-3ac3-49de-91e9-6eac2647b59a-kube-api-access-svf6r\") pod \"maas-api-revoke-keys-e2e-praxis-a-389c4e-966lm\" (UID: \"db133a39-3ac3-49de-91e9-6eac2647b59a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-389c4e-966lm" Oct 30 20:14:05.409392 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:05.409263 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db133a39-3ac3-49de-91e9-6eac2647b59a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-389c4e-966lm\" (UID: \"db133a39-3ac3-49de-91e9-6eac2647b59a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-389c4e-966lm" Oct 30 20:14:05.409392 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:05.409336 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db133a39-3ac3-49de-91e9-6eac2647b59a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-389c4e-966lm\" (UID: \"db133a39-3ac3-49de-91e9-6eac2647b59a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-389c4e-966lm" Oct 30 20:14:05.409959 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:05.409929 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db133a39-3ac3-49de-91e9-6eac2647b59a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-389c4e-966lm\" (UID: \"db133a39-3ac3-49de-91e9-6eac2647b59a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-389c4e-966lm" Oct 30 20:14:05.411637 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:05.411615 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db133a39-3ac3-49de-91e9-6eac2647b59a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-389c4e-966lm\" (UID: \"db133a39-3ac3-49de-91e9-6eac2647b59a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-389c4e-966lm" Oct 30 20:14:05.471704 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:05.471681 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svf6r\" (UniqueName: \"kubernetes.io/projected/db133a39-3ac3-49de-91e9-6eac2647b59a-kube-api-access-svf6r\") pod \"maas-api-revoke-keys-e2e-praxis-a-389c4e-966lm\" (UID: \"db133a39-3ac3-49de-91e9-6eac2647b59a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-389c4e-966lm" Oct 30 20:14:05.539573 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:05.539548 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21aab5d1-2ph6l" Oct 30 20:14:05.539573 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:05.539559 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21aab5d1-2ph6l" event={"ID":"d8cedf1d-e992-4152-9f8f-2ba60cadc3be","Type":"ContainerDied","Data":"7f2b02cfa2186986d336c8b5779e5c136fe64092ecf0ca4bcb9b113535258a73"} Oct 30 20:14:05.539735 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:05.539601 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f2b02cfa2186986d336c8b5779e5c136fe64092ecf0ca4bcb9b113535258a73" Oct 30 20:14:05.578217 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:05.578187 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-389c4e-966lm" Oct 30 20:14:05.953939 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:05.953913 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-389c4e-966lm"] Oct 30 20:14:05.955465 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:14:05.955439 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb133a39_3ac3_49de_91e9_6eac2647b59a.slice/crio-1029e5a2609651a7e8f953bb3119c7c2c453b9da02352d8c449eeddcc43ef2f7 WatchSource:0}: Error finding container 1029e5a2609651a7e8f953bb3119c7c2c453b9da02352d8c449eeddcc43ef2f7: Status 404 returned error can't find the container with id 1029e5a2609651a7e8f953bb3119c7c2c453b9da02352d8c449eeddcc43ef2f7 Oct 30 20:14:06.546779 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:06.546739 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-389c4e-966lm" event={"ID":"db133a39-3ac3-49de-91e9-6eac2647b59a","Type":"ContainerStarted","Data":"e014890f34708b21ae4762d5e290b22d8d22e67fb2fe6c689d0595120e647fe3"} Oct 30 20:14:06.546967 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:06.546803 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-389c4e-966lm" event={"ID":"db133a39-3ac3-49de-91e9-6eac2647b59a","Type":"ContainerStarted","Data":"1029e5a2609651a7e8f953bb3119c7c2c453b9da02352d8c449eeddcc43ef2f7"} Oct 30 20:14:06.666100 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:06.665954 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-389c4e-966lm" podStartSLOduration=1.665938138 podStartE2EDuration="1.665938138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:14:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:14:06.664667275 +0000 UTC m=+2950.182982748" watchObservedRunningTime="2026-10-30 20:14:06.665938138 +0000 UTC m=+2950.184253597" Oct 30 20:14:07.555528 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:07.555495 4292 generic.go:356] "Generic (PLEG): container finished" podID="db133a39-3ac3-49de-91e9-6eac2647b59a" containerID="e014890f34708b21ae4762d5e290b22d8d22e67fb2fe6c689d0595120e647fe3" exitCode=0 Oct 30 20:14:07.555934 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:07.555568 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-389c4e-966lm" event={"ID":"db133a39-3ac3-49de-91e9-6eac2647b59a","Type":"ContainerDied","Data":"e014890f34708b21ae4762d5e290b22d8d22e67fb2fe6c689d0595120e647fe3"} Oct 30 20:14:08.712336 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:08.712304 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-389c4e-966lm" Oct 30 20:14:08.745764 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:08.745720 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db133a39-3ac3-49de-91e9-6eac2647b59a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db133a39-3ac3-49de-91e9-6eac2647b59a-maas-api-openshift-ca\") pod \"db133a39-3ac3-49de-91e9-6eac2647b59a\" (UID: \"db133a39-3ac3-49de-91e9-6eac2647b59a\") " Oct 30 20:14:08.745915 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:08.745779 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db133a39-3ac3-49de-91e9-6eac2647b59a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db133a39-3ac3-49de-91e9-6eac2647b59a-maas-api-xks-ca\") pod \"db133a39-3ac3-49de-91e9-6eac2647b59a\" (UID: \"db133a39-3ac3-49de-91e9-6eac2647b59a\") " Oct 30 20:14:08.745959 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:08.745920 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db133a39-3ac3-49de-91e9-6eac2647b59a-kube-api-access-svf6r\" (UniqueName: \"kubernetes.io/projected/db133a39-3ac3-49de-91e9-6eac2647b59a-kube-api-access-svf6r\") pod \"db133a39-3ac3-49de-91e9-6eac2647b59a\" (UID: \"db133a39-3ac3-49de-91e9-6eac2647b59a\") " Oct 30 20:14:08.746218 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:08.746184 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db133a39-3ac3-49de-91e9-6eac2647b59a-maas-api-openshift-ca" pod "db133a39-3ac3-49de-91e9-6eac2647b59a" (UID: "db133a39-3ac3-49de-91e9-6eac2647b59a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:14:08.747852 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:08.747827 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db133a39-3ac3-49de-91e9-6eac2647b59a-maas-api-xks-ca" pod "db133a39-3ac3-49de-91e9-6eac2647b59a" (UID: "db133a39-3ac3-49de-91e9-6eac2647b59a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:14:08.747993 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:08.747973 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db133a39-3ac3-49de-91e9-6eac2647b59a-kube-api-access-svf6r" pod "db133a39-3ac3-49de-91e9-6eac2647b59a" (UID: "db133a39-3ac3-49de-91e9-6eac2647b59a"). InnerVolumeSpecName "kube-api-access-svf6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:08.846937 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:08.846844 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svf6r\" (UniqueName: \"kubernetes.io/projected/db133a39-3ac3-49de-91e9-6eac2647b59a-kube-api-access-svf6r\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:08.846937 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:08.846875 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db133a39-3ac3-49de-91e9-6eac2647b59a-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:08.846937 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:08.846888 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db133a39-3ac3-49de-91e9-6eac2647b59a-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:09.205012 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:09.204934 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h"] Oct 30 20:14:09.205217 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:09.205191 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h" podUID="607ab3e3-4a07-41ef-b1d9-983ddc238675" containerName="istio-proxy" containerID="cri-o://b397e12c0686dbb626c951e53d9345ec67cfb80586e67be88595f77d3d4b4358" gracePeriod=30 Oct 30 20:14:09.567636 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:09.567609 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-389c4e-966lm" Oct 30 20:14:09.567847 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:09.567646 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-389c4e-966lm" event={"ID":"db133a39-3ac3-49de-91e9-6eac2647b59a","Type":"ContainerDied","Data":"1029e5a2609651a7e8f953bb3119c7c2c453b9da02352d8c449eeddcc43ef2f7"} Oct 30 20:14:09.567847 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:09.567693 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1029e5a2609651a7e8f953bb3119c7c2c453b9da02352d8c449eeddcc43ef2f7" Oct 30 20:14:11.750424 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:11.750391 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca6a94a9-vqwph"] Oct 30 20:14:11.750852 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:11.750844 4292 cpu_manager.go:414] "containerMap: removing container" podUID="db133a39-3ac3-49de-91e9-6eac2647b59a" containerName="revoke-keys" Oct 30 20:14:11.750901 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:11.750856 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db133a39-3ac3-49de-91e9-6eac2647b59a" containerName="revoke-keys" Oct 30 20:14:11.750948 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:11.750937 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="db133a39-3ac3-49de-91e9-6eac2647b59a" containerName="revoke-keys" Oct 30 20:14:11.754389 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:11.754345 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca6a94a9-vqwph" Oct 30 20:14:11.780066 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:11.780041 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a9a2029-1c42-4e15-8c85-f3bc00ff4536-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca6a94a9-vqwph\" (UID: \"3a9a2029-1c42-4e15-8c85-f3bc00ff4536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca6a94a9-vqwph" Oct 30 20:14:11.780234 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:11.780206 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a9a2029-1c42-4e15-8c85-f3bc00ff4536-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca6a94a9-vqwph\" (UID: \"3a9a2029-1c42-4e15-8c85-f3bc00ff4536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca6a94a9-vqwph" Oct 30 20:14:11.780319 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:11.780282 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkzm5\" (UniqueName: \"kubernetes.io/projected/3a9a2029-1c42-4e15-8c85-f3bc00ff4536-kube-api-access-lkzm5\") pod \"maas-api-revoke-sub-ca6a94a9-vqwph\" (UID: \"3a9a2029-1c42-4e15-8c85-f3bc00ff4536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca6a94a9-vqwph" Oct 30 20:14:11.811120 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:11.811090 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca6a94a9-vqwph"] Oct 30 20:14:11.857387 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:11.857345 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:14:11.857577 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:11.857473 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\"" Oct 30 20:14:11.881481 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:11.881445 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a9a2029-1c42-4e15-8c85-f3bc00ff4536-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca6a94a9-vqwph\" (UID: \"3a9a2029-1c42-4e15-8c85-f3bc00ff4536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca6a94a9-vqwph" Oct 30 20:14:11.881652 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:11.881599 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkzm5\" (UniqueName: \"kubernetes.io/projected/3a9a2029-1c42-4e15-8c85-f3bc00ff4536-kube-api-access-lkzm5\") pod \"maas-api-revoke-sub-ca6a94a9-vqwph\" (UID: \"3a9a2029-1c42-4e15-8c85-f3bc00ff4536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca6a94a9-vqwph" Oct 30 20:14:11.881748 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:11.881717 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a9a2029-1c42-4e15-8c85-f3bc00ff4536-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca6a94a9-vqwph\" (UID: \"3a9a2029-1c42-4e15-8c85-f3bc00ff4536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca6a94a9-vqwph" Oct 30 20:14:11.882287 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:11.882262 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a9a2029-1c42-4e15-8c85-f3bc00ff4536-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca6a94a9-vqwph\" (UID: \"3a9a2029-1c42-4e15-8c85-f3bc00ff4536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca6a94a9-vqwph" Oct 30 20:14:11.884223 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:11.884196 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a9a2029-1c42-4e15-8c85-f3bc00ff4536-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca6a94a9-vqwph\" (UID: \"3a9a2029-1c42-4e15-8c85-f3bc00ff4536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca6a94a9-vqwph" Oct 30 20:14:11.945703 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:11.945662 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkzm5\" (UniqueName: \"kubernetes.io/projected/3a9a2029-1c42-4e15-8c85-f3bc00ff4536-kube-api-access-lkzm5\") pod \"maas-api-revoke-sub-ca6a94a9-vqwph\" (UID: \"3a9a2029-1c42-4e15-8c85-f3bc00ff4536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca6a94a9-vqwph" Oct 30 20:14:12.071241 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:12.071209 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca6a94a9-vqwph" Oct 30 20:14:12.456679 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:12.456647 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca6a94a9-vqwph"] Oct 30 20:14:12.457285 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:14:12.457247 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a9a2029_1c42_4e15_8c85_f3bc00ff4536.slice/crio-afca05d20d4b9e7d614517e3b008fd40db658b4f61e39e820d1d5dc38795e146 WatchSource:0}: Error finding container afca05d20d4b9e7d614517e3b008fd40db658b4f61e39e820d1d5dc38795e146: Status 404 returned error can't find the container with id afca05d20d4b9e7d614517e3b008fd40db658b4f61e39e820d1d5dc38795e146 Oct 30 20:14:12.585809 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:12.585770 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca6a94a9-vqwph" event={"ID":"3a9a2029-1c42-4e15-8c85-f3bc00ff4536","Type":"ContainerStarted","Data":"afca05d20d4b9e7d614517e3b008fd40db658b4f61e39e820d1d5dc38795e146"} Oct 30 20:14:13.220712 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:14:13.220678 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:14:13.593295 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:13.593245 4292 generic.go:356] "Generic (PLEG): container finished" podID="3a9a2029-1c42-4e15-8c85-f3bc00ff4536" containerID="bf6543046839a2af1c0e490df5185eca3def9309b080ac91522516b19a88c7d2" exitCode=0 Oct 30 20:14:13.593480 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:13.593327 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca6a94a9-vqwph" event={"ID":"3a9a2029-1c42-4e15-8c85-f3bc00ff4536","Type":"ContainerDied","Data":"bf6543046839a2af1c0e490df5185eca3def9309b080ac91522516b19a88c7d2"} Oct 30 20:14:13.639449 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:13.639416 4292 patch_prober.go:29] interesting pod/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:14:13.639598 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:13.639475 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h" podUID="607ab3e3-4a07-41ef-b1d9-983ddc238675" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:14:14.460299 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:14.460276 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h" Oct 30 20:14:14.522034 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:14.522006 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/607ab3e3-4a07-41ef-b1d9-983ddc238675-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/607ab3e3-4a07-41ef-b1d9-983ddc238675-credential-socket\") pod \"607ab3e3-4a07-41ef-b1d9-983ddc238675\" (UID: \"607ab3e3-4a07-41ef-b1d9-983ddc238675\") " Oct 30 20:14:14.522223 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:14.522051 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/607ab3e3-4a07-41ef-b1d9-983ddc238675-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/607ab3e3-4a07-41ef-b1d9-983ddc238675-workload-certs\") pod \"607ab3e3-4a07-41ef-b1d9-983ddc238675\" (UID: \"607ab3e3-4a07-41ef-b1d9-983ddc238675\") " Oct 30 20:14:14.522223 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:14.522072 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/607ab3e3-4a07-41ef-b1d9-983ddc238675-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/607ab3e3-4a07-41ef-b1d9-983ddc238675-istiod-ca-cert\") pod \"607ab3e3-4a07-41ef-b1d9-983ddc238675\" (UID: \"607ab3e3-4a07-41ef-b1d9-983ddc238675\") " Oct 30 20:14:14.522223 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:14.522125 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/607ab3e3-4a07-41ef-b1d9-983ddc238675-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/607ab3e3-4a07-41ef-b1d9-983ddc238675-istio-envoy\") pod \"607ab3e3-4a07-41ef-b1d9-983ddc238675\" (UID: \"607ab3e3-4a07-41ef-b1d9-983ddc238675\") " Oct 30 20:14:14.522223 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:14.522169 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/607ab3e3-4a07-41ef-b1d9-983ddc238675-istio-token\" (UniqueName: \"kubernetes.io/projected/607ab3e3-4a07-41ef-b1d9-983ddc238675-istio-token\") pod \"607ab3e3-4a07-41ef-b1d9-983ddc238675\" (UID: \"607ab3e3-4a07-41ef-b1d9-983ddc238675\") " Oct 30 20:14:14.522445 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:14.522344 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/607ab3e3-4a07-41ef-b1d9-983ddc238675-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/607ab3e3-4a07-41ef-b1d9-983ddc238675-istio-data\") pod \"607ab3e3-4a07-41ef-b1d9-983ddc238675\" (UID: \"607ab3e3-4a07-41ef-b1d9-983ddc238675\") " Oct 30 20:14:14.522445 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:14.522425 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/607ab3e3-4a07-41ef-b1d9-983ddc238675-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/607ab3e3-4a07-41ef-b1d9-983ddc238675-workload-socket\") pod \"607ab3e3-4a07-41ef-b1d9-983ddc238675\" (UID: \"607ab3e3-4a07-41ef-b1d9-983ddc238675\") " Oct 30 20:14:14.522582 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:14.522564 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/607ab3e3-4a07-41ef-b1d9-983ddc238675-kube-api-access-bc28h\" (UniqueName: \"kubernetes.io/projected/607ab3e3-4a07-41ef-b1d9-983ddc238675-kube-api-access-bc28h\") pod \"607ab3e3-4a07-41ef-b1d9-983ddc238675\" (UID: \"607ab3e3-4a07-41ef-b1d9-983ddc238675\") " Oct 30 20:14:14.522637 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:14.522604 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/607ab3e3-4a07-41ef-b1d9-983ddc238675-workload-certs" pod "607ab3e3-4a07-41ef-b1d9-983ddc238675" (UID: "607ab3e3-4a07-41ef-b1d9-983ddc238675"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:14.522637 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:14.522608 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/607ab3e3-4a07-41ef-b1d9-983ddc238675-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/607ab3e3-4a07-41ef-b1d9-983ddc238675-istio-podinfo\") pod \"607ab3e3-4a07-41ef-b1d9-983ddc238675\" (UID: \"607ab3e3-4a07-41ef-b1d9-983ddc238675\") " Oct 30 20:14:14.522786 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:14.522762 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/607ab3e3-4a07-41ef-b1d9-983ddc238675-istio-data" pod "607ab3e3-4a07-41ef-b1d9-983ddc238675" (UID: "607ab3e3-4a07-41ef-b1d9-983ddc238675"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:14.522857 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:14.522826 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/607ab3e3-4a07-41ef-b1d9-983ddc238675-istiod-ca-cert" pod "607ab3e3-4a07-41ef-b1d9-983ddc238675" (UID: "607ab3e3-4a07-41ef-b1d9-983ddc238675"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:14:14.522917 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:14.522864 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/607ab3e3-4a07-41ef-b1d9-983ddc238675-workload-socket" pod "607ab3e3-4a07-41ef-b1d9-983ddc238675" (UID: "607ab3e3-4a07-41ef-b1d9-983ddc238675"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:14.523124 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:14.523099 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/607ab3e3-4a07-41ef-b1d9-983ddc238675-credential-socket" pod "607ab3e3-4a07-41ef-b1d9-983ddc238675" (UID: "607ab3e3-4a07-41ef-b1d9-983ddc238675"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:14.523272 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:14.523176 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/607ab3e3-4a07-41ef-b1d9-983ddc238675-workload-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:14.523272 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:14.523198 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/607ab3e3-4a07-41ef-b1d9-983ddc238675-istiod-ca-cert\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:14.523272 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:14.523211 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/607ab3e3-4a07-41ef-b1d9-983ddc238675-istio-data\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:14.523272 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:14.523225 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/607ab3e3-4a07-41ef-b1d9-983ddc238675-workload-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:14.524941 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:14.524915 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/607ab3e3-4a07-41ef-b1d9-983ddc238675-kube-api-access-bc28h" pod "607ab3e3-4a07-41ef-b1d9-983ddc238675" (UID: "607ab3e3-4a07-41ef-b1d9-983ddc238675"). InnerVolumeSpecName "kube-api-access-bc28h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:14.525066 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:14.525017 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/607ab3e3-4a07-41ef-b1d9-983ddc238675-istio-token" pod "607ab3e3-4a07-41ef-b1d9-983ddc238675" (UID: "607ab3e3-4a07-41ef-b1d9-983ddc238675"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:14.525204 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:14.525185 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/607ab3e3-4a07-41ef-b1d9-983ddc238675-istio-podinfo" pod "607ab3e3-4a07-41ef-b1d9-983ddc238675" (UID: "607ab3e3-4a07-41ef-b1d9-983ddc238675"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:14:14.525313 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:14.525294 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/607ab3e3-4a07-41ef-b1d9-983ddc238675-istio-envoy" pod "607ab3e3-4a07-41ef-b1d9-983ddc238675" (UID: "607ab3e3-4a07-41ef-b1d9-983ddc238675"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:14.602770 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:14.602738 4292 generic.go:356] "Generic (PLEG): container finished" podID="607ab3e3-4a07-41ef-b1d9-983ddc238675" containerID="b397e12c0686dbb626c951e53d9345ec67cfb80586e67be88595f77d3d4b4358" exitCode=0 Oct 30 20:14:14.602914 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:14.602799 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h" event={"ID":"607ab3e3-4a07-41ef-b1d9-983ddc238675","Type":"ContainerDied","Data":"b397e12c0686dbb626c951e53d9345ec67cfb80586e67be88595f77d3d4b4358"} Oct 30 20:14:14.602914 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:14.602839 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h" Oct 30 20:14:14.602914 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:14.602850 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h" event={"ID":"607ab3e3-4a07-41ef-b1d9-983ddc238675","Type":"ContainerDied","Data":"c9b644e11c33545163a21af537811a24df1df026e5b2b5678c169a21f0736b43"} Oct 30 20:14:14.602914 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:14.602878 4292 scope.go:122] "RemoveContainer" containerID="b397e12c0686dbb626c951e53d9345ec67cfb80586e67be88595f77d3d4b4358" Oct 30 20:14:14.612673 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:14.612653 4292 scope.go:122] "RemoveContainer" containerID="de5d656416bcad3602c679675ec8e66f99ba27958850ecdb72f7fe4c58a312bd" Oct 30 20:14:14.621992 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:14.621970 4292 scope.go:122] "RemoveContainer" containerID="b397e12c0686dbb626c951e53d9345ec67cfb80586e67be88595f77d3d4b4358" Oct 30 20:14:14.622286 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:14:14.622266 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b397e12c0686dbb626c951e53d9345ec67cfb80586e67be88595f77d3d4b4358\": container with ID starting with b397e12c0686dbb626c951e53d9345ec67cfb80586e67be88595f77d3d4b4358 not found: ID does not exist" containerID="b397e12c0686dbb626c951e53d9345ec67cfb80586e67be88595f77d3d4b4358" Oct 30 20:14:14.622396 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:14.622294 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b397e12c0686dbb626c951e53d9345ec67cfb80586e67be88595f77d3d4b4358"} err="failed to get container status \"b397e12c0686dbb626c951e53d9345ec67cfb80586e67be88595f77d3d4b4358\": rpc error: code = NotFound desc = could not find container \"b397e12c0686dbb626c951e53d9345ec67cfb80586e67be88595f77d3d4b4358\": container with ID starting with b397e12c0686dbb626c951e53d9345ec67cfb80586e67be88595f77d3d4b4358 not found: ID does not exist" Oct 30 20:14:14.622396 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:14.622316 4292 scope.go:122] "RemoveContainer" containerID="de5d656416bcad3602c679675ec8e66f99ba27958850ecdb72f7fe4c58a312bd" Oct 30 20:14:14.622700 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:14:14.622677 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de5d656416bcad3602c679675ec8e66f99ba27958850ecdb72f7fe4c58a312bd\": container with ID starting with de5d656416bcad3602c679675ec8e66f99ba27958850ecdb72f7fe4c58a312bd not found: ID does not exist" containerID="de5d656416bcad3602c679675ec8e66f99ba27958850ecdb72f7fe4c58a312bd" Oct 30 20:14:14.622766 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:14.622707 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de5d656416bcad3602c679675ec8e66f99ba27958850ecdb72f7fe4c58a312bd"} err="failed to get container status \"de5d656416bcad3602c679675ec8e66f99ba27958850ecdb72f7fe4c58a312bd\": rpc error: code = NotFound desc = could not find container \"de5d656416bcad3602c679675ec8e66f99ba27958850ecdb72f7fe4c58a312bd\": container with ID starting with de5d656416bcad3602c679675ec8e66f99ba27958850ecdb72f7fe4c58a312bd not found: ID does not exist" Oct 30 20:14:14.624272 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:14.624246 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bc28h\" (UniqueName: \"kubernetes.io/projected/607ab3e3-4a07-41ef-b1d9-983ddc238675-kube-api-access-bc28h\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:14.624272 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:14.624269 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/607ab3e3-4a07-41ef-b1d9-983ddc238675-istio-podinfo\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:14.624272 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:14.624279 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/607ab3e3-4a07-41ef-b1d9-983ddc238675-credential-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:14.624528 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:14.624290 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/607ab3e3-4a07-41ef-b1d9-983ddc238675-istio-envoy\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:14.624528 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:14.624299 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/607ab3e3-4a07-41ef-b1d9-983ddc238675-istio-token\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:14.740720 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:14.740693 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca6a94a9-vqwph" Oct 30 20:14:14.757813 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:14.757777 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h"] Oct 30 20:14:14.762186 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:14.762164 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-c60685-openshift-default-68f76c87c-b2w7h"] Oct 30 20:14:14.826162 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:14.826098 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a9a2029-1c42-4e15-8c85-f3bc00ff4536-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a9a2029-1c42-4e15-8c85-f3bc00ff4536-maas-api-xks-ca\") pod \"3a9a2029-1c42-4e15-8c85-f3bc00ff4536\" (UID: \"3a9a2029-1c42-4e15-8c85-f3bc00ff4536\") " Oct 30 20:14:14.826162 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:14.826140 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a9a2029-1c42-4e15-8c85-f3bc00ff4536-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a9a2029-1c42-4e15-8c85-f3bc00ff4536-maas-api-openshift-ca\") pod \"3a9a2029-1c42-4e15-8c85-f3bc00ff4536\" (UID: \"3a9a2029-1c42-4e15-8c85-f3bc00ff4536\") " Oct 30 20:14:14.826341 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:14.826178 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a9a2029-1c42-4e15-8c85-f3bc00ff4536-kube-api-access-lkzm5\" (UniqueName: \"kubernetes.io/projected/3a9a2029-1c42-4e15-8c85-f3bc00ff4536-kube-api-access-lkzm5\") pod \"3a9a2029-1c42-4e15-8c85-f3bc00ff4536\" (UID: \"3a9a2029-1c42-4e15-8c85-f3bc00ff4536\") " Oct 30 20:14:14.826535 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:14.826513 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a9a2029-1c42-4e15-8c85-f3bc00ff4536-maas-api-openshift-ca" pod "3a9a2029-1c42-4e15-8c85-f3bc00ff4536" (UID: "3a9a2029-1c42-4e15-8c85-f3bc00ff4536"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:14:14.828251 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:14.828220 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a9a2029-1c42-4e15-8c85-f3bc00ff4536-maas-api-xks-ca" pod "3a9a2029-1c42-4e15-8c85-f3bc00ff4536" (UID: "3a9a2029-1c42-4e15-8c85-f3bc00ff4536"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:14:14.828251 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:14.828239 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a9a2029-1c42-4e15-8c85-f3bc00ff4536-kube-api-access-lkzm5" pod "3a9a2029-1c42-4e15-8c85-f3bc00ff4536" (UID: "3a9a2029-1c42-4e15-8c85-f3bc00ff4536"). InnerVolumeSpecName "kube-api-access-lkzm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:14.927630 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:14.927599 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a9a2029-1c42-4e15-8c85-f3bc00ff4536-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:14.927630 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:14.927626 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a9a2029-1c42-4e15-8c85-f3bc00ff4536-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:14.927630 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:14.927638 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkzm5\" (UniqueName: \"kubernetes.io/projected/3a9a2029-1c42-4e15-8c85-f3bc00ff4536-kube-api-access-lkzm5\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:15.496404 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:15.496312 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d371e629-v8hpp"] Oct 30 20:14:15.496991 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:15.496973 4292 cpu_manager.go:414] "containerMap: removing container" podUID="607ab3e3-4a07-41ef-b1d9-983ddc238675" containerName="istio-proxy" Oct 30 20:14:15.497031 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:15.496994 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="607ab3e3-4a07-41ef-b1d9-983ddc238675" containerName="istio-proxy" Oct 30 20:14:15.497031 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:15.497026 4292 cpu_manager.go:414] "containerMap: removing container" podUID="3a9a2029-1c42-4e15-8c85-f3bc00ff4536" containerName="revoke-keys" Oct 30 20:14:15.497098 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:15.497035 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a9a2029-1c42-4e15-8c85-f3bc00ff4536" containerName="revoke-keys" Oct 30 20:14:15.497150 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:15.497139 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="607ab3e3-4a07-41ef-b1d9-983ddc238675" containerName="istio-proxy" Oct 30 20:14:15.497189 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:15.497155 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="607ab3e3-4a07-41ef-b1d9-983ddc238675" containerName="istio-proxy" Oct 30 20:14:15.497189 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:15.497165 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a9a2029-1c42-4e15-8c85-f3bc00ff4536" containerName="revoke-keys" Oct 30 20:14:15.500435 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:15.500416 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d371e629-v8hpp" Oct 30 20:14:15.554771 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:15.554737 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d371e629-v8hpp"] Oct 30 20:14:15.610303 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:15.610270 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca6a94a9-vqwph" event={"ID":"3a9a2029-1c42-4e15-8c85-f3bc00ff4536","Type":"ContainerDied","Data":"afca05d20d4b9e7d614517e3b008fd40db658b4f61e39e820d1d5dc38795e146"} Oct 30 20:14:15.610303 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:15.610295 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca6a94a9-vqwph" Oct 30 20:14:15.610498 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:15.610315 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afca05d20d4b9e7d614517e3b008fd40db658b4f61e39e820d1d5dc38795e146" Oct 30 20:14:15.634956 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:15.634923 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pc77\" (UniqueName: \"kubernetes.io/projected/65bb082c-b874-4da9-9d10-8e42f6de6576-kube-api-access-9pc77\") pod \"maas-api-revoke-sub-d371e629-v8hpp\" (UID: \"65bb082c-b874-4da9-9d10-8e42f6de6576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d371e629-v8hpp" Oct 30 20:14:15.635083 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:15.634971 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65bb082c-b874-4da9-9d10-8e42f6de6576-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d371e629-v8hpp\" (UID: \"65bb082c-b874-4da9-9d10-8e42f6de6576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d371e629-v8hpp" Oct 30 20:14:15.635083 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:15.635052 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65bb082c-b874-4da9-9d10-8e42f6de6576-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d371e629-v8hpp\" (UID: \"65bb082c-b874-4da9-9d10-8e42f6de6576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d371e629-v8hpp" Oct 30 20:14:15.736142 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:15.736112 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pc77\" (UniqueName: \"kubernetes.io/projected/65bb082c-b874-4da9-9d10-8e42f6de6576-kube-api-access-9pc77\") pod \"maas-api-revoke-sub-d371e629-v8hpp\" (UID: \"65bb082c-b874-4da9-9d10-8e42f6de6576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d371e629-v8hpp" Oct 30 20:14:15.736293 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:15.736152 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65bb082c-b874-4da9-9d10-8e42f6de6576-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d371e629-v8hpp\" (UID: \"65bb082c-b874-4da9-9d10-8e42f6de6576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d371e629-v8hpp" Oct 30 20:14:15.736293 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:15.736183 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65bb082c-b874-4da9-9d10-8e42f6de6576-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d371e629-v8hpp\" (UID: \"65bb082c-b874-4da9-9d10-8e42f6de6576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d371e629-v8hpp" Oct 30 20:14:15.736866 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:15.736848 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65bb082c-b874-4da9-9d10-8e42f6de6576-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d371e629-v8hpp\" (UID: \"65bb082c-b874-4da9-9d10-8e42f6de6576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d371e629-v8hpp" Oct 30 20:14:15.738552 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:15.738524 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65bb082c-b874-4da9-9d10-8e42f6de6576-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d371e629-v8hpp\" (UID: \"65bb082c-b874-4da9-9d10-8e42f6de6576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d371e629-v8hpp" Oct 30 20:14:15.798713 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:15.798688 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pc77\" (UniqueName: \"kubernetes.io/projected/65bb082c-b874-4da9-9d10-8e42f6de6576-kube-api-access-9pc77\") pod \"maas-api-revoke-sub-d371e629-v8hpp\" (UID: \"65bb082c-b874-4da9-9d10-8e42f6de6576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d371e629-v8hpp" Oct 30 20:14:15.813521 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:15.813501 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d371e629-v8hpp" Oct 30 20:14:16.195756 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:16.195732 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d371e629-v8hpp"] Oct 30 20:14:16.197078 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:14:16.197051 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65bb082c_b874_4da9_9d10_8e42f6de6576.slice/crio-d0c2e271386e1d99fb3c0ea6822a825ff749e9f0438fe4e7c0f709786ff3a9f1 WatchSource:0}: Error finding container d0c2e271386e1d99fb3c0ea6822a825ff749e9f0438fe4e7c0f709786ff3a9f1: Status 404 returned error can't find the container with id d0c2e271386e1d99fb3c0ea6822a825ff749e9f0438fe4e7c0f709786ff3a9f1 Oct 30 20:14:16.582531 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:16.582497 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="607ab3e3-4a07-41ef-b1d9-983ddc238675" path="/var/lib/kubelet/pods/607ab3e3-4a07-41ef-b1d9-983ddc238675/volumes" Oct 30 20:14:16.617149 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:16.617115 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d371e629-v8hpp" event={"ID":"65bb082c-b874-4da9-9d10-8e42f6de6576","Type":"ContainerStarted","Data":"191aab6c03a43670954b81a54b15dc9db6040eec65405f66f4b5bb28d3543d19"} Oct 30 20:14:16.617321 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:16.617167 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d371e629-v8hpp" event={"ID":"65bb082c-b874-4da9-9d10-8e42f6de6576","Type":"ContainerStarted","Data":"d0c2e271386e1d99fb3c0ea6822a825ff749e9f0438fe4e7c0f709786ff3a9f1"} Oct 30 20:14:16.738898 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:16.738799 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d371e629-v8hpp" podStartSLOduration=1.738784822 podStartE2EDuration="1.738784822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:14:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:14:16.736493583 +0000 UTC m=+2960.254809040" watchObservedRunningTime="2026-10-30 20:14:16.738784822 +0000 UTC m=+2960.257100296" Oct 30 20:14:17.628195 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:17.628151 4292 generic.go:356] "Generic (PLEG): container finished" podID="65bb082c-b874-4da9-9d10-8e42f6de6576" containerID="191aab6c03a43670954b81a54b15dc9db6040eec65405f66f4b5bb28d3543d19" exitCode=0 Oct 30 20:14:17.628715 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:17.628266 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d371e629-v8hpp" event={"ID":"65bb082c-b874-4da9-9d10-8e42f6de6576","Type":"ContainerDied","Data":"191aab6c03a43670954b81a54b15dc9db6040eec65405f66f4b5bb28d3543d19"} Oct 30 20:14:18.839720 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:18.839696 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d371e629-v8hpp" Oct 30 20:14:18.973113 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:18.973033 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65bb082c-b874-4da9-9d10-8e42f6de6576-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65bb082c-b874-4da9-9d10-8e42f6de6576-maas-api-xks-ca\") pod \"65bb082c-b874-4da9-9d10-8e42f6de6576\" (UID: \"65bb082c-b874-4da9-9d10-8e42f6de6576\") " Oct 30 20:14:18.973113 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:18.973087 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65bb082c-b874-4da9-9d10-8e42f6de6576-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65bb082c-b874-4da9-9d10-8e42f6de6576-maas-api-openshift-ca\") pod \"65bb082c-b874-4da9-9d10-8e42f6de6576\" (UID: \"65bb082c-b874-4da9-9d10-8e42f6de6576\") " Oct 30 20:14:18.973344 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:18.973174 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65bb082c-b874-4da9-9d10-8e42f6de6576-kube-api-access-9pc77\" (UniqueName: \"kubernetes.io/projected/65bb082c-b874-4da9-9d10-8e42f6de6576-kube-api-access-9pc77\") pod \"65bb082c-b874-4da9-9d10-8e42f6de6576\" (UID: \"65bb082c-b874-4da9-9d10-8e42f6de6576\") " Oct 30 20:14:18.973457 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:18.973432 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65bb082c-b874-4da9-9d10-8e42f6de6576-maas-api-openshift-ca" pod "65bb082c-b874-4da9-9d10-8e42f6de6576" (UID: "65bb082c-b874-4da9-9d10-8e42f6de6576"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:14:18.973643 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:18.973623 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65bb082c-b874-4da9-9d10-8e42f6de6576-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:18.975290 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:18.975268 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65bb082c-b874-4da9-9d10-8e42f6de6576-maas-api-xks-ca" pod "65bb082c-b874-4da9-9d10-8e42f6de6576" (UID: "65bb082c-b874-4da9-9d10-8e42f6de6576"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:14:18.975567 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:18.975537 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65bb082c-b874-4da9-9d10-8e42f6de6576-kube-api-access-9pc77" pod "65bb082c-b874-4da9-9d10-8e42f6de6576" (UID: "65bb082c-b874-4da9-9d10-8e42f6de6576"). InnerVolumeSpecName "kube-api-access-9pc77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:19.074745 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:19.074718 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65bb082c-b874-4da9-9d10-8e42f6de6576-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:19.074745 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:19.074743 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pc77\" (UniqueName: \"kubernetes.io/projected/65bb082c-b874-4da9-9d10-8e42f6de6576-kube-api-access-9pc77\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:19.649178 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:19.649143 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d371e629-v8hpp" event={"ID":"65bb082c-b874-4da9-9d10-8e42f6de6576","Type":"ContainerDied","Data":"d0c2e271386e1d99fb3c0ea6822a825ff749e9f0438fe4e7c0f709786ff3a9f1"} Oct 30 20:14:19.649178 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:19.649175 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d371e629-v8hpp" Oct 30 20:14:19.649422 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:19.649201 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0c2e271386e1d99fb3c0ea6822a825ff749e9f0438fe4e7c0f709786ff3a9f1" Oct 30 20:14:22.295940 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:22.295905 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45fc6b7f-hz7dl"] Oct 30 20:14:22.296377 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:22.296341 4292 cpu_manager.go:414] "containerMap: removing container" podUID="607ab3e3-4a07-41ef-b1d9-983ddc238675" containerName="istio-proxy" Oct 30 20:14:22.296377 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:22.296367 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="607ab3e3-4a07-41ef-b1d9-983ddc238675" containerName="istio-proxy" Oct 30 20:14:22.296468 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:22.296382 4292 cpu_manager.go:414] "containerMap: removing container" podUID="65bb082c-b874-4da9-9d10-8e42f6de6576" containerName="revoke-keys" Oct 30 20:14:22.296468 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:22.296387 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65bb082c-b874-4da9-9d10-8e42f6de6576" containerName="revoke-keys" Oct 30 20:14:22.296468 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:22.296457 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="65bb082c-b874-4da9-9d10-8e42f6de6576" containerName="revoke-keys" Oct 30 20:14:22.299832 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:22.299815 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45fc6b7f-hz7dl" Oct 30 20:14:22.357553 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:22.357523 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45fc6b7f-hz7dl"] Oct 30 20:14:22.404237 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:22.404208 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:14:22.404447 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:22.404328 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\"" Oct 30 20:14:22.411128 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:22.411098 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05331ed1-fa17-4d42-81a7-9626c610a70c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45fc6b7f-hz7dl\" (UID: \"05331ed1-fa17-4d42-81a7-9626c610a70c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45fc6b7f-hz7dl" Oct 30 20:14:22.411335 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:22.411316 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05331ed1-fa17-4d42-81a7-9626c610a70c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45fc6b7f-hz7dl\" (UID: \"05331ed1-fa17-4d42-81a7-9626c610a70c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45fc6b7f-hz7dl" Oct 30 20:14:22.411431 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:22.411401 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqkg9\" (UniqueName: \"kubernetes.io/projected/05331ed1-fa17-4d42-81a7-9626c610a70c-kube-api-access-pqkg9\") pod \"maas-api-revoke-sub-45fc6b7f-hz7dl\" (UID: \"05331ed1-fa17-4d42-81a7-9626c610a70c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45fc6b7f-hz7dl" Oct 30 20:14:22.512838 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:22.512807 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05331ed1-fa17-4d42-81a7-9626c610a70c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45fc6b7f-hz7dl\" (UID: \"05331ed1-fa17-4d42-81a7-9626c610a70c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45fc6b7f-hz7dl" Oct 30 20:14:22.512838 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:22.512837 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqkg9\" (UniqueName: \"kubernetes.io/projected/05331ed1-fa17-4d42-81a7-9626c610a70c-kube-api-access-pqkg9\") pod \"maas-api-revoke-sub-45fc6b7f-hz7dl\" (UID: \"05331ed1-fa17-4d42-81a7-9626c610a70c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45fc6b7f-hz7dl" Oct 30 20:14:22.513019 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:22.512868 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05331ed1-fa17-4d42-81a7-9626c610a70c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45fc6b7f-hz7dl\" (UID: \"05331ed1-fa17-4d42-81a7-9626c610a70c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45fc6b7f-hz7dl" Oct 30 20:14:22.513581 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:22.513561 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05331ed1-fa17-4d42-81a7-9626c610a70c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45fc6b7f-hz7dl\" (UID: \"05331ed1-fa17-4d42-81a7-9626c610a70c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45fc6b7f-hz7dl" Oct 30 20:14:22.515172 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:22.515155 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05331ed1-fa17-4d42-81a7-9626c610a70c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45fc6b7f-hz7dl\" (UID: \"05331ed1-fa17-4d42-81a7-9626c610a70c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45fc6b7f-hz7dl" Oct 30 20:14:22.587379 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:22.587283 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqkg9\" (UniqueName: \"kubernetes.io/projected/05331ed1-fa17-4d42-81a7-9626c610a70c-kube-api-access-pqkg9\") pod \"maas-api-revoke-sub-45fc6b7f-hz7dl\" (UID: \"05331ed1-fa17-4d42-81a7-9626c610a70c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45fc6b7f-hz7dl" Oct 30 20:14:22.613432 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:22.613406 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45fc6b7f-hz7dl" Oct 30 20:14:22.993624 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:22.993602 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45fc6b7f-hz7dl"] Oct 30 20:14:22.995731 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:14:22.995704 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05331ed1_fa17_4d42_81a7_9626c610a70c.slice/crio-200055c15d352cbde817673ccb8846b897b9dea01ce331eca116518d1474cd2e WatchSource:0}: Error finding container 200055c15d352cbde817673ccb8846b897b9dea01ce331eca116518d1474cd2e: Status 404 returned error can't find the container with id 200055c15d352cbde817673ccb8846b897b9dea01ce331eca116518d1474cd2e Oct 30 20:14:23.121075 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:23.121042 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e6b14d-mztks"] Oct 30 20:14:23.125568 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:23.125542 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e6b14d-mztks"] Oct 30 20:14:23.674918 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:23.674883 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45fc6b7f-hz7dl" event={"ID":"05331ed1-fa17-4d42-81a7-9626c610a70c","Type":"ContainerStarted","Data":"b9eaa0190d6e29bb0abce812f2c9cac9586a608cd0e39cd82cbb934dc04ccc9a"} Oct 30 20:14:23.675323 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:23.674944 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45fc6b7f-hz7dl" event={"ID":"05331ed1-fa17-4d42-81a7-9626c610a70c","Type":"ContainerStarted","Data":"200055c15d352cbde817673ccb8846b897b9dea01ce331eca116518d1474cd2e"} Oct 30 20:14:23.815138 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:23.815086 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45fc6b7f-hz7dl" podStartSLOduration=1.815072669 podStartE2EDuration="1.815072669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:14:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:14:23.811693734 +0000 UTC m=+2967.330009204" watchObservedRunningTime="2026-10-30 20:14:23.815072669 +0000 UTC m=+2967.333388127" Oct 30 20:14:24.581730 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:24.581695 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ffe6f78-11ec-4efd-aba7-30411430be0d" path="/var/lib/kubelet/pods/1ffe6f78-11ec-4efd-aba7-30411430be0d/volumes" Oct 30 20:14:24.681545 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:24.681506 4292 generic.go:356] "Generic (PLEG): container finished" podID="05331ed1-fa17-4d42-81a7-9626c610a70c" containerID="b9eaa0190d6e29bb0abce812f2c9cac9586a608cd0e39cd82cbb934dc04ccc9a" exitCode=0 Oct 30 20:14:24.681962 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:24.681587 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45fc6b7f-hz7dl" event={"ID":"05331ed1-fa17-4d42-81a7-9626c610a70c","Type":"ContainerDied","Data":"b9eaa0190d6e29bb0abce812f2c9cac9586a608cd0e39cd82cbb934dc04ccc9a"} Oct 30 20:14:25.838201 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:25.838177 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45fc6b7f-hz7dl" Oct 30 20:14:25.952701 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:25.952616 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05331ed1-fa17-4d42-81a7-9626c610a70c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05331ed1-fa17-4d42-81a7-9626c610a70c-maas-api-openshift-ca\") pod \"05331ed1-fa17-4d42-81a7-9626c610a70c\" (UID: \"05331ed1-fa17-4d42-81a7-9626c610a70c\") " Oct 30 20:14:25.952701 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:25.952678 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05331ed1-fa17-4d42-81a7-9626c610a70c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05331ed1-fa17-4d42-81a7-9626c610a70c-maas-api-xks-ca\") pod \"05331ed1-fa17-4d42-81a7-9626c610a70c\" (UID: \"05331ed1-fa17-4d42-81a7-9626c610a70c\") " Oct 30 20:14:25.952701 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:25.952701 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05331ed1-fa17-4d42-81a7-9626c610a70c-kube-api-access-pqkg9\" (UniqueName: \"kubernetes.io/projected/05331ed1-fa17-4d42-81a7-9626c610a70c-kube-api-access-pqkg9\") pod \"05331ed1-fa17-4d42-81a7-9626c610a70c\" (UID: \"05331ed1-fa17-4d42-81a7-9626c610a70c\") " Oct 30 20:14:25.953029 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:25.953006 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05331ed1-fa17-4d42-81a7-9626c610a70c-maas-api-openshift-ca" pod "05331ed1-fa17-4d42-81a7-9626c610a70c" (UID: "05331ed1-fa17-4d42-81a7-9626c610a70c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:14:25.954814 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:25.954792 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05331ed1-fa17-4d42-81a7-9626c610a70c-kube-api-access-pqkg9" pod "05331ed1-fa17-4d42-81a7-9626c610a70c" (UID: "05331ed1-fa17-4d42-81a7-9626c610a70c"). InnerVolumeSpecName "kube-api-access-pqkg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:25.954900 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:25.954874 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05331ed1-fa17-4d42-81a7-9626c610a70c-maas-api-xks-ca" pod "05331ed1-fa17-4d42-81a7-9626c610a70c" (UID: "05331ed1-fa17-4d42-81a7-9626c610a70c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:14:26.053736 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:26.053701 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05331ed1-fa17-4d42-81a7-9626c610a70c-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:26.053736 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:26.053730 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05331ed1-fa17-4d42-81a7-9626c610a70c-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:26.053925 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:26.053743 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqkg9\" (UniqueName: \"kubernetes.io/projected/05331ed1-fa17-4d42-81a7-9626c610a70c-kube-api-access-pqkg9\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:26.695541 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:26.695502 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45fc6b7f-hz7dl" event={"ID":"05331ed1-fa17-4d42-81a7-9626c610a70c","Type":"ContainerDied","Data":"200055c15d352cbde817673ccb8846b897b9dea01ce331eca116518d1474cd2e"} Oct 30 20:14:26.695761 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:26.695543 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45fc6b7f-hz7dl" Oct 30 20:14:26.695761 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:26.695558 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="200055c15d352cbde817673ccb8846b897b9dea01ce331eca116518d1474cd2e" Oct 30 20:14:28.275635 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:28.275602 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3584be9e-wwcnp"] Oct 30 20:14:28.276061 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:28.276020 4292 cpu_manager.go:414] "containerMap: removing container" podUID="05331ed1-fa17-4d42-81a7-9626c610a70c" containerName="revoke-keys" Oct 30 20:14:28.276061 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:28.276030 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05331ed1-fa17-4d42-81a7-9626c610a70c" containerName="revoke-keys" Oct 30 20:14:28.276179 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:28.276117 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="05331ed1-fa17-4d42-81a7-9626c610a70c" containerName="revoke-keys" Oct 30 20:14:28.279313 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:28.279297 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3584be9e-wwcnp" Oct 30 20:14:28.335374 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:28.335309 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3584be9e-wwcnp"] Oct 30 20:14:28.377916 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:28.376128 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c408b28f-6739-418e-89f7-f693f17f7d34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3584be9e-wwcnp\" (UID: \"c408b28f-6739-418e-89f7-f693f17f7d34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3584be9e-wwcnp" Oct 30 20:14:28.377916 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:28.376215 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsg7n\" (UniqueName: \"kubernetes.io/projected/c408b28f-6739-418e-89f7-f693f17f7d34-kube-api-access-tsg7n\") pod \"maas-api-revoke-sub-3584be9e-wwcnp\" (UID: \"c408b28f-6739-418e-89f7-f693f17f7d34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3584be9e-wwcnp" Oct 30 20:14:28.377916 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:28.376326 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c408b28f-6739-418e-89f7-f693f17f7d34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3584be9e-wwcnp\" (UID: \"c408b28f-6739-418e-89f7-f693f17f7d34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3584be9e-wwcnp" Oct 30 20:14:28.391659 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:28.385224 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:14:28.391659 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:28.388951 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\"" Oct 30 20:14:28.477973 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:28.477934 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsg7n\" (UniqueName: \"kubernetes.io/projected/c408b28f-6739-418e-89f7-f693f17f7d34-kube-api-access-tsg7n\") pod \"maas-api-revoke-sub-3584be9e-wwcnp\" (UID: \"c408b28f-6739-418e-89f7-f693f17f7d34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3584be9e-wwcnp" Oct 30 20:14:28.478152 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:28.478073 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c408b28f-6739-418e-89f7-f693f17f7d34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3584be9e-wwcnp\" (UID: \"c408b28f-6739-418e-89f7-f693f17f7d34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3584be9e-wwcnp" Oct 30 20:14:28.478238 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:28.478215 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c408b28f-6739-418e-89f7-f693f17f7d34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3584be9e-wwcnp\" (UID: \"c408b28f-6739-418e-89f7-f693f17f7d34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3584be9e-wwcnp" Oct 30 20:14:28.478877 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:28.478853 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c408b28f-6739-418e-89f7-f693f17f7d34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3584be9e-wwcnp\" (UID: \"c408b28f-6739-418e-89f7-f693f17f7d34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3584be9e-wwcnp" Oct 30 20:14:28.480681 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:28.480661 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c408b28f-6739-418e-89f7-f693f17f7d34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3584be9e-wwcnp\" (UID: \"c408b28f-6739-418e-89f7-f693f17f7d34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3584be9e-wwcnp" Oct 30 20:14:28.540291 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:28.540233 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsg7n\" (UniqueName: \"kubernetes.io/projected/c408b28f-6739-418e-89f7-f693f17f7d34-kube-api-access-tsg7n\") pod \"maas-api-revoke-sub-3584be9e-wwcnp\" (UID: \"c408b28f-6739-418e-89f7-f693f17f7d34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3584be9e-wwcnp" Oct 30 20:14:28.591768 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:28.591737 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3584be9e-wwcnp" Oct 30 20:14:28.977408 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:28.977381 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3584be9e-wwcnp"] Oct 30 20:14:28.979862 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:14:28.979832 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc408b28f_6739_418e_89f7_f693f17f7d34.slice/crio-e8cd72df83a2fc729520150f25605789e31c2076da6045a0a42e21fe7057d141 WatchSource:0}: Error finding container e8cd72df83a2fc729520150f25605789e31c2076da6045a0a42e21fe7057d141: Status 404 returned error can't find the container with id e8cd72df83a2fc729520150f25605789e31c2076da6045a0a42e21fe7057d141 Oct 30 20:14:29.715831 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:29.715787 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3584be9e-wwcnp" event={"ID":"c408b28f-6739-418e-89f7-f693f17f7d34","Type":"ContainerStarted","Data":"114e270aa59d72f17a93bd1990eb4a6f415f8826baf866143ae1bd3dad077fd1"} Oct 30 20:14:29.716286 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:29.715846 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3584be9e-wwcnp" event={"ID":"c408b28f-6739-418e-89f7-f693f17f7d34","Type":"ContainerStarted","Data":"e8cd72df83a2fc729520150f25605789e31c2076da6045a0a42e21fe7057d141"} Oct 30 20:14:29.833920 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:29.833873 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3584be9e-wwcnp" podStartSLOduration=1.833859127 podStartE2EDuration="1.833859127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:14:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:14:29.832536728 +0000 UTC m=+2973.350852186" watchObservedRunningTime="2026-10-30 20:14:29.833859127 +0000 UTC m=+2973.352174589" Oct 30 20:14:30.723127 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:30.723093 4292 generic.go:356] "Generic (PLEG): container finished" podID="c408b28f-6739-418e-89f7-f693f17f7d34" containerID="114e270aa59d72f17a93bd1990eb4a6f415f8826baf866143ae1bd3dad077fd1" exitCode=0 Oct 30 20:14:30.723631 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:30.723172 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3584be9e-wwcnp" event={"ID":"c408b28f-6739-418e-89f7-f693f17f7d34","Type":"ContainerDied","Data":"114e270aa59d72f17a93bd1990eb4a6f415f8826baf866143ae1bd3dad077fd1"} Oct 30 20:14:31.609095 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:31.609061 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5c6b996-8nhtz"] Oct 30 20:14:31.612928 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:31.612910 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5c6b996-8nhtz" Oct 30 20:14:31.668305 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:31.668271 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5c6b996-8nhtz"] Oct 30 20:14:31.722373 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:31.722315 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67n2q\" (UniqueName: \"kubernetes.io/projected/89c7d9dc-a73b-418a-bb1d-37aab103b116-kube-api-access-67n2q\") pod \"maas-api-revoke-sub-a5c6b996-8nhtz\" (UID: \"89c7d9dc-a73b-418a-bb1d-37aab103b116\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5c6b996-8nhtz" Oct 30 20:14:31.722634 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:31.722606 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89c7d9dc-a73b-418a-bb1d-37aab103b116-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5c6b996-8nhtz\" (UID: \"89c7d9dc-a73b-418a-bb1d-37aab103b116\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5c6b996-8nhtz" Oct 30 20:14:31.722766 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:31.722648 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89c7d9dc-a73b-418a-bb1d-37aab103b116-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5c6b996-8nhtz\" (UID: \"89c7d9dc-a73b-418a-bb1d-37aab103b116\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5c6b996-8nhtz" Oct 30 20:14:31.824148 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:31.824107 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89c7d9dc-a73b-418a-bb1d-37aab103b116-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5c6b996-8nhtz\" (UID: \"89c7d9dc-a73b-418a-bb1d-37aab103b116\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5c6b996-8nhtz" Oct 30 20:14:31.824604 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:31.824159 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89c7d9dc-a73b-418a-bb1d-37aab103b116-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5c6b996-8nhtz\" (UID: \"89c7d9dc-a73b-418a-bb1d-37aab103b116\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5c6b996-8nhtz" Oct 30 20:14:31.824604 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:31.824212 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67n2q\" (UniqueName: \"kubernetes.io/projected/89c7d9dc-a73b-418a-bb1d-37aab103b116-kube-api-access-67n2q\") pod \"maas-api-revoke-sub-a5c6b996-8nhtz\" (UID: \"89c7d9dc-a73b-418a-bb1d-37aab103b116\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5c6b996-8nhtz" Oct 30 20:14:31.824910 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:31.824884 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89c7d9dc-a73b-418a-bb1d-37aab103b116-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5c6b996-8nhtz\" (UID: \"89c7d9dc-a73b-418a-bb1d-37aab103b116\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5c6b996-8nhtz" Oct 30 20:14:31.826714 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:31.826684 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89c7d9dc-a73b-418a-bb1d-37aab103b116-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5c6b996-8nhtz\" (UID: \"89c7d9dc-a73b-418a-bb1d-37aab103b116\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5c6b996-8nhtz" Oct 30 20:14:31.865094 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:31.865036 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3584be9e-wwcnp" Oct 30 20:14:31.886867 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:31.886836 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67n2q\" (UniqueName: \"kubernetes.io/projected/89c7d9dc-a73b-418a-bb1d-37aab103b116-kube-api-access-67n2q\") pod \"maas-api-revoke-sub-a5c6b996-8nhtz\" (UID: \"89c7d9dc-a73b-418a-bb1d-37aab103b116\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5c6b996-8nhtz" Oct 30 20:14:31.925670 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:31.925643 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5c6b996-8nhtz" Oct 30 20:14:32.025646 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:32.025618 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c408b28f-6739-418e-89f7-f693f17f7d34-kube-api-access-tsg7n\" (UniqueName: \"kubernetes.io/projected/c408b28f-6739-418e-89f7-f693f17f7d34-kube-api-access-tsg7n\") pod \"c408b28f-6739-418e-89f7-f693f17f7d34\" (UID: \"c408b28f-6739-418e-89f7-f693f17f7d34\") " Oct 30 20:14:32.025807 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:32.025731 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c408b28f-6739-418e-89f7-f693f17f7d34-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c408b28f-6739-418e-89f7-f693f17f7d34-maas-api-xks-ca\") pod \"c408b28f-6739-418e-89f7-f693f17f7d34\" (UID: \"c408b28f-6739-418e-89f7-f693f17f7d34\") " Oct 30 20:14:32.025855 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:32.025815 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c408b28f-6739-418e-89f7-f693f17f7d34-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c408b28f-6739-418e-89f7-f693f17f7d34-maas-api-openshift-ca\") pod \"c408b28f-6739-418e-89f7-f693f17f7d34\" (UID: \"c408b28f-6739-418e-89f7-f693f17f7d34\") " Oct 30 20:14:32.026325 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:32.026295 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c408b28f-6739-418e-89f7-f693f17f7d34-maas-api-openshift-ca" pod "c408b28f-6739-418e-89f7-f693f17f7d34" (UID: "c408b28f-6739-418e-89f7-f693f17f7d34"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:14:32.028015 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:32.027995 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c408b28f-6739-418e-89f7-f693f17f7d34-maas-api-xks-ca" pod "c408b28f-6739-418e-89f7-f693f17f7d34" (UID: "c408b28f-6739-418e-89f7-f693f17f7d34"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:14:32.028291 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:32.028267 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c408b28f-6739-418e-89f7-f693f17f7d34-kube-api-access-tsg7n" pod "c408b28f-6739-418e-89f7-f693f17f7d34" (UID: "c408b28f-6739-418e-89f7-f693f17f7d34"). InnerVolumeSpecName "kube-api-access-tsg7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:32.127570 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:32.127490 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c408b28f-6739-418e-89f7-f693f17f7d34-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:32.127570 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:32.127529 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c408b28f-6739-418e-89f7-f693f17f7d34-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:32.127570 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:32.127545 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsg7n\" (UniqueName: \"kubernetes.io/projected/c408b28f-6739-418e-89f7-f693f17f7d34-kube-api-access-tsg7n\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:32.299769 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:32.299742 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5c6b996-8nhtz"] Oct 30 20:14:32.300821 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:14:32.300792 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89c7d9dc_a73b_418a_bb1d_37aab103b116.slice/crio-a9524c60aa0cd2447aa1ba920e658dfbc5e8be8447d697600cbfe78052b18a6b WatchSource:0}: Error finding container a9524c60aa0cd2447aa1ba920e658dfbc5e8be8447d697600cbfe78052b18a6b: Status 404 returned error can't find the container with id a9524c60aa0cd2447aa1ba920e658dfbc5e8be8447d697600cbfe78052b18a6b Oct 30 20:14:32.737538 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:32.737457 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3584be9e-wwcnp" Oct 30 20:14:32.737538 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:32.737477 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3584be9e-wwcnp" event={"ID":"c408b28f-6739-418e-89f7-f693f17f7d34","Type":"ContainerDied","Data":"e8cd72df83a2fc729520150f25605789e31c2076da6045a0a42e21fe7057d141"} Oct 30 20:14:32.737538 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:32.737530 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8cd72df83a2fc729520150f25605789e31c2076da6045a0a42e21fe7057d141" Oct 30 20:14:32.739222 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:32.739192 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5c6b996-8nhtz" event={"ID":"89c7d9dc-a73b-418a-bb1d-37aab103b116","Type":"ContainerStarted","Data":"216ee02f2849761a8cd78bfe00d702a451d09a409fb97edddf6f2355095c0a1e"} Oct 30 20:14:32.739381 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:32.739243 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5c6b996-8nhtz" event={"ID":"89c7d9dc-a73b-418a-bb1d-37aab103b116","Type":"ContainerStarted","Data":"a9524c60aa0cd2447aa1ba920e658dfbc5e8be8447d697600cbfe78052b18a6b"} Oct 30 20:14:32.863456 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:32.863404 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5c6b996-8nhtz" podStartSLOduration=1.863389076 podStartE2EDuration="1.863389076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:14:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:14:32.861759317 +0000 UTC m=+2976.380074776" watchObservedRunningTime="2026-10-30 20:14:32.863389076 +0000 UTC m=+2976.381704530" Oct 30 20:14:33.750536 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:33.750500 4292 generic.go:356] "Generic (PLEG): container finished" podID="89c7d9dc-a73b-418a-bb1d-37aab103b116" containerID="216ee02f2849761a8cd78bfe00d702a451d09a409fb97edddf6f2355095c0a1e" exitCode=0 Oct 30 20:14:33.750708 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:33.750597 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5c6b996-8nhtz" event={"ID":"89c7d9dc-a73b-418a-bb1d-37aab103b116","Type":"ContainerDied","Data":"216ee02f2849761a8cd78bfe00d702a451d09a409fb97edddf6f2355095c0a1e"} Oct 30 20:14:34.899072 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:34.899045 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5c6b996-8nhtz" Oct 30 20:14:35.058510 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:35.058474 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89c7d9dc-a73b-418a-bb1d-37aab103b116-kube-api-access-67n2q\" (UniqueName: \"kubernetes.io/projected/89c7d9dc-a73b-418a-bb1d-37aab103b116-kube-api-access-67n2q\") pod \"89c7d9dc-a73b-418a-bb1d-37aab103b116\" (UID: \"89c7d9dc-a73b-418a-bb1d-37aab103b116\") " Oct 30 20:14:35.058682 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:35.058575 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89c7d9dc-a73b-418a-bb1d-37aab103b116-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89c7d9dc-a73b-418a-bb1d-37aab103b116-maas-api-xks-ca\") pod \"89c7d9dc-a73b-418a-bb1d-37aab103b116\" (UID: \"89c7d9dc-a73b-418a-bb1d-37aab103b116\") " Oct 30 20:14:35.058682 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:35.058640 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89c7d9dc-a73b-418a-bb1d-37aab103b116-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89c7d9dc-a73b-418a-bb1d-37aab103b116-maas-api-openshift-ca\") pod \"89c7d9dc-a73b-418a-bb1d-37aab103b116\" (UID: \"89c7d9dc-a73b-418a-bb1d-37aab103b116\") " Oct 30 20:14:35.059005 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:35.058970 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89c7d9dc-a73b-418a-bb1d-37aab103b116-maas-api-openshift-ca" pod "89c7d9dc-a73b-418a-bb1d-37aab103b116" (UID: "89c7d9dc-a73b-418a-bb1d-37aab103b116"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:14:35.059146 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:35.059122 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89c7d9dc-a73b-418a-bb1d-37aab103b116-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:35.060526 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:35.060507 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89c7d9dc-a73b-418a-bb1d-37aab103b116-kube-api-access-67n2q" pod "89c7d9dc-a73b-418a-bb1d-37aab103b116" (UID: "89c7d9dc-a73b-418a-bb1d-37aab103b116"). InnerVolumeSpecName "kube-api-access-67n2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:35.060674 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:35.060660 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89c7d9dc-a73b-418a-bb1d-37aab103b116-maas-api-xks-ca" pod "89c7d9dc-a73b-418a-bb1d-37aab103b116" (UID: "89c7d9dc-a73b-418a-bb1d-37aab103b116"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:14:35.160681 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:35.160644 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67n2q\" (UniqueName: \"kubernetes.io/projected/89c7d9dc-a73b-418a-bb1d-37aab103b116-kube-api-access-67n2q\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:35.160681 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:35.160675 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89c7d9dc-a73b-418a-bb1d-37aab103b116-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:35.765917 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:35.765887 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5c6b996-8nhtz" Oct 30 20:14:35.766078 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:35.765916 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5c6b996-8nhtz" event={"ID":"89c7d9dc-a73b-418a-bb1d-37aab103b116","Type":"ContainerDied","Data":"a9524c60aa0cd2447aa1ba920e658dfbc5e8be8447d697600cbfe78052b18a6b"} Oct 30 20:14:35.766078 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:35.765977 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9524c60aa0cd2447aa1ba920e658dfbc5e8be8447d697600cbfe78052b18a6b" Oct 30 20:14:39.057332 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:39.057298 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8e1c336-tdkjm"] Oct 30 20:14:39.057775 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:39.057760 4292 cpu_manager.go:414] "containerMap: removing container" podUID="89c7d9dc-a73b-418a-bb1d-37aab103b116" containerName="revoke-keys" Oct 30 20:14:39.057821 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:39.057777 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89c7d9dc-a73b-418a-bb1d-37aab103b116" containerName="revoke-keys" Oct 30 20:14:39.057821 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:39.057805 4292 cpu_manager.go:414] "containerMap: removing container" podUID="c408b28f-6739-418e-89f7-f693f17f7d34" containerName="revoke-keys" Oct 30 20:14:39.057821 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:39.057811 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c408b28f-6739-418e-89f7-f693f17f7d34" containerName="revoke-keys" Oct 30 20:14:39.057916 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:39.057871 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="c408b28f-6739-418e-89f7-f693f17f7d34" containerName="revoke-keys" Oct 30 20:14:39.057916 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:39.057880 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="89c7d9dc-a73b-418a-bb1d-37aab103b116" containerName="revoke-keys" Oct 30 20:14:39.061514 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:39.061492 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8e1c336-tdkjm" Oct 30 20:14:39.116589 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:39.116561 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8e1c336-tdkjm"] Oct 30 20:14:39.163482 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:39.163447 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:14:39.163634 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:39.163499 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\"" Oct 30 20:14:39.211752 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:39.211723 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/432c0a5c-9c82-4d91-abe6-189b090f17d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8e1c336-tdkjm\" (UID: \"432c0a5c-9c82-4d91-abe6-189b090f17d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8e1c336-tdkjm" Oct 30 20:14:39.211908 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:39.211888 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/432c0a5c-9c82-4d91-abe6-189b090f17d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8e1c336-tdkjm\" (UID: \"432c0a5c-9c82-4d91-abe6-189b090f17d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8e1c336-tdkjm" Oct 30 20:14:39.211997 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:39.211983 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh7wc\" (UniqueName: \"kubernetes.io/projected/432c0a5c-9c82-4d91-abe6-189b090f17d9-kube-api-access-zh7wc\") pod \"maas-api-revoke-sub-e8e1c336-tdkjm\" (UID: \"432c0a5c-9c82-4d91-abe6-189b090f17d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8e1c336-tdkjm" Oct 30 20:14:39.313668 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:39.313598 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/432c0a5c-9c82-4d91-abe6-189b090f17d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8e1c336-tdkjm\" (UID: \"432c0a5c-9c82-4d91-abe6-189b090f17d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8e1c336-tdkjm" Oct 30 20:14:39.313804 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:39.313680 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh7wc\" (UniqueName: \"kubernetes.io/projected/432c0a5c-9c82-4d91-abe6-189b090f17d9-kube-api-access-zh7wc\") pod \"maas-api-revoke-sub-e8e1c336-tdkjm\" (UID: \"432c0a5c-9c82-4d91-abe6-189b090f17d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8e1c336-tdkjm" Oct 30 20:14:39.313804 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:39.313714 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/432c0a5c-9c82-4d91-abe6-189b090f17d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8e1c336-tdkjm\" (UID: \"432c0a5c-9c82-4d91-abe6-189b090f17d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8e1c336-tdkjm" Oct 30 20:14:39.314316 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:39.314294 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/432c0a5c-9c82-4d91-abe6-189b090f17d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8e1c336-tdkjm\" (UID: \"432c0a5c-9c82-4d91-abe6-189b090f17d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8e1c336-tdkjm" Oct 30 20:14:39.316050 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:39.316029 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/432c0a5c-9c82-4d91-abe6-189b090f17d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8e1c336-tdkjm\" (UID: \"432c0a5c-9c82-4d91-abe6-189b090f17d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8e1c336-tdkjm" Oct 30 20:14:39.378140 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:39.378107 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh7wc\" (UniqueName: \"kubernetes.io/projected/432c0a5c-9c82-4d91-abe6-189b090f17d9-kube-api-access-zh7wc\") pod \"maas-api-revoke-sub-e8e1c336-tdkjm\" (UID: \"432c0a5c-9c82-4d91-abe6-189b090f17d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8e1c336-tdkjm" Oct 30 20:14:39.675410 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:39.675330 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8e1c336-tdkjm" Oct 30 20:14:40.066009 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:40.065985 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8e1c336-tdkjm"] Oct 30 20:14:40.067794 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:14:40.067763 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod432c0a5c_9c82_4d91_abe6_189b090f17d9.slice/crio-4d6c57899a6a90b1fd26ac44e363eda9b577d11f756aa465b4684c8219a29046 WatchSource:0}: Error finding container 4d6c57899a6a90b1fd26ac44e363eda9b577d11f756aa465b4684c8219a29046: Status 404 returned error can't find the container with id 4d6c57899a6a90b1fd26ac44e363eda9b577d11f756aa465b4684c8219a29046 Oct 30 20:14:40.796626 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:40.796578 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8e1c336-tdkjm" event={"ID":"432c0a5c-9c82-4d91-abe6-189b090f17d9","Type":"ContainerStarted","Data":"210f784ac02bce26bf133a8714416bbee316a11a20d58fd306d9a0616da61b5d"} Oct 30 20:14:40.796817 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:40.796669 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8e1c336-tdkjm" event={"ID":"432c0a5c-9c82-4d91-abe6-189b090f17d9","Type":"ContainerStarted","Data":"4d6c57899a6a90b1fd26ac44e363eda9b577d11f756aa465b4684c8219a29046"} Oct 30 20:14:40.916186 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:40.916134 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8e1c336-tdkjm" podStartSLOduration=1.916119942 podStartE2EDuration="1.916119942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:14:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:14:40.915403061 +0000 UTC m=+2984.433718519" watchObservedRunningTime="2026-10-30 20:14:40.916119942 +0000 UTC m=+2984.434435450" Oct 30 20:14:41.803126 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:41.803083 4292 generic.go:356] "Generic (PLEG): container finished" podID="432c0a5c-9c82-4d91-abe6-189b090f17d9" containerID="210f784ac02bce26bf133a8714416bbee316a11a20d58fd306d9a0616da61b5d" exitCode=0 Oct 30 20:14:41.803504 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:41.803172 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8e1c336-tdkjm" event={"ID":"432c0a5c-9c82-4d91-abe6-189b090f17d9","Type":"ContainerDied","Data":"210f784ac02bce26bf133a8714416bbee316a11a20d58fd306d9a0616da61b5d"} Oct 30 20:14:42.617054 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:42.617026 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-04679c-86c54d7f84-prhpr"] Oct 30 20:14:42.729557 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:42.729522 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-04679c-696cc7bfd7-zdpv7"] Oct 30 20:14:42.950994 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:42.950965 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8e1c336-tdkjm" Oct 30 20:14:43.148112 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:43.148084 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/432c0a5c-9c82-4d91-abe6-189b090f17d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/432c0a5c-9c82-4d91-abe6-189b090f17d9-maas-api-xks-ca\") pod \"432c0a5c-9c82-4d91-abe6-189b090f17d9\" (UID: \"432c0a5c-9c82-4d91-abe6-189b090f17d9\") " Oct 30 20:14:43.148279 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:43.148223 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/432c0a5c-9c82-4d91-abe6-189b090f17d9-kube-api-access-zh7wc\" (UniqueName: \"kubernetes.io/projected/432c0a5c-9c82-4d91-abe6-189b090f17d9-kube-api-access-zh7wc\") pod \"432c0a5c-9c82-4d91-abe6-189b090f17d9\" (UID: \"432c0a5c-9c82-4d91-abe6-189b090f17d9\") " Oct 30 20:14:43.148279 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:43.148253 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/432c0a5c-9c82-4d91-abe6-189b090f17d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/432c0a5c-9c82-4d91-abe6-189b090f17d9-maas-api-openshift-ca\") pod \"432c0a5c-9c82-4d91-abe6-189b090f17d9\" (UID: \"432c0a5c-9c82-4d91-abe6-189b090f17d9\") " Oct 30 20:14:43.148648 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:43.148620 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/432c0a5c-9c82-4d91-abe6-189b090f17d9-maas-api-openshift-ca" pod "432c0a5c-9c82-4d91-abe6-189b090f17d9" (UID: "432c0a5c-9c82-4d91-abe6-189b090f17d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:14:43.150158 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:43.150137 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/432c0a5c-9c82-4d91-abe6-189b090f17d9-maas-api-xks-ca" pod "432c0a5c-9c82-4d91-abe6-189b090f17d9" (UID: "432c0a5c-9c82-4d91-abe6-189b090f17d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:14:43.150573 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:43.150554 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/432c0a5c-9c82-4d91-abe6-189b090f17d9-kube-api-access-zh7wc" pod "432c0a5c-9c82-4d91-abe6-189b090f17d9" (UID: "432c0a5c-9c82-4d91-abe6-189b090f17d9"). InnerVolumeSpecName "kube-api-access-zh7wc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:43.219855 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:14:43.219772 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:14:43.249522 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:43.249495 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh7wc\" (UniqueName: \"kubernetes.io/projected/432c0a5c-9c82-4d91-abe6-189b090f17d9-kube-api-access-zh7wc\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:43.249522 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:43.249519 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/432c0a5c-9c82-4d91-abe6-189b090f17d9-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:43.249694 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:43.249530 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/432c0a5c-9c82-4d91-abe6-189b090f17d9-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:43.817072 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:43.817036 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8e1c336-tdkjm" event={"ID":"432c0a5c-9c82-4d91-abe6-189b090f17d9","Type":"ContainerDied","Data":"4d6c57899a6a90b1fd26ac44e363eda9b577d11f756aa465b4684c8219a29046"} Oct 30 20:14:43.817260 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:43.817096 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d6c57899a6a90b1fd26ac44e363eda9b577d11f756aa465b4684c8219a29046" Oct 30 20:14:43.817260 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:43.817045 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8e1c336-tdkjm" Oct 30 20:14:47.617898 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:47.617839 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-04679c-86c54d7f84-prhpr" podUID="191cdc99-91e9-49de-9006-66e40aa246b6" containerName="payload-processing" containerID="cri-o://96dd42c675b063c1f91ab2385392830cdaf9832d9e83ebefc10981e5640605df" gracePeriod=25 Oct 30 20:14:47.730671 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:47.730620 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-04679c-696cc7bfd7-zdpv7" podUID="6dee7e19-6aeb-46a7-b763-52cbd3cf1113" containerName="payload-pre-processing" containerID="cri-o://3dbf1983887a6ed07df40468dbfd9096f7b13f2b7dba14318e7a86f03a6a7025" gracePeriod=25 Oct 30 20:14:47.842799 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:47.842767 4292 generic.go:356] "Generic (PLEG): container finished" podID="191cdc99-91e9-49de-9006-66e40aa246b6" containerID="96dd42c675b063c1f91ab2385392830cdaf9832d9e83ebefc10981e5640605df" exitCode=0 Oct 30 20:14:47.842994 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:47.842843 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-04679c-86c54d7f84-prhpr" event={"ID":"191cdc99-91e9-49de-9006-66e40aa246b6","Type":"ContainerDied","Data":"96dd42c675b063c1f91ab2385392830cdaf9832d9e83ebefc10981e5640605df"} Oct 30 20:14:47.889094 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:47.889034 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-04679c-86c54d7f84-prhpr" Oct 30 20:14:47.996557 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:47.996524 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/191cdc99-91e9-49de-9006-66e40aa246b6-kube-api-access-bj55v\" (UniqueName: \"kubernetes.io/projected/191cdc99-91e9-49de-9006-66e40aa246b6-kube-api-access-bj55v\") pod \"191cdc99-91e9-49de-9006-66e40aa246b6\" (UID: \"191cdc99-91e9-49de-9006-66e40aa246b6\") " Oct 30 20:14:47.996700 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:47.996574 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/191cdc99-91e9-49de-9006-66e40aa246b6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/191cdc99-91e9-49de-9006-66e40aa246b6-plugins-config-volume\") pod \"191cdc99-91e9-49de-9006-66e40aa246b6\" (UID: \"191cdc99-91e9-49de-9006-66e40aa246b6\") " Oct 30 20:14:47.996948 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:47.996926 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/191cdc99-91e9-49de-9006-66e40aa246b6-plugins-config-volume" pod "191cdc99-91e9-49de-9006-66e40aa246b6" (UID: "191cdc99-91e9-49de-9006-66e40aa246b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:14:47.997602 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:47.997582 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-04679c-696cc7bfd7-zdpv7" Oct 30 20:14:47.998637 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:47.998622 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/191cdc99-91e9-49de-9006-66e40aa246b6-kube-api-access-bj55v" pod "191cdc99-91e9-49de-9006-66e40aa246b6" (UID: "191cdc99-91e9-49de-9006-66e40aa246b6"). InnerVolumeSpecName "kube-api-access-bj55v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:48.097732 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:48.097700 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dee7e19-6aeb-46a7-b763-52cbd3cf1113-kube-api-access-rs52q\" (UniqueName: \"kubernetes.io/projected/6dee7e19-6aeb-46a7-b763-52cbd3cf1113-kube-api-access-rs52q\") pod \"6dee7e19-6aeb-46a7-b763-52cbd3cf1113\" (UID: \"6dee7e19-6aeb-46a7-b763-52cbd3cf1113\") " Oct 30 20:14:48.097923 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:48.097828 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dee7e19-6aeb-46a7-b763-52cbd3cf1113-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dee7e19-6aeb-46a7-b763-52cbd3cf1113-plugins-config-volume\") pod \"6dee7e19-6aeb-46a7-b763-52cbd3cf1113\" (UID: \"6dee7e19-6aeb-46a7-b763-52cbd3cf1113\") " Oct 30 20:14:48.098024 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:48.098010 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bj55v\" (UniqueName: \"kubernetes.io/projected/191cdc99-91e9-49de-9006-66e40aa246b6-kube-api-access-bj55v\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:48.098087 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:48.098024 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/191cdc99-91e9-49de-9006-66e40aa246b6-plugins-config-volume\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:48.098228 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:48.098205 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dee7e19-6aeb-46a7-b763-52cbd3cf1113-plugins-config-volume" pod "6dee7e19-6aeb-46a7-b763-52cbd3cf1113" (UID: "6dee7e19-6aeb-46a7-b763-52cbd3cf1113"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:14:48.099828 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:48.099804 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dee7e19-6aeb-46a7-b763-52cbd3cf1113-kube-api-access-rs52q" pod "6dee7e19-6aeb-46a7-b763-52cbd3cf1113" (UID: "6dee7e19-6aeb-46a7-b763-52cbd3cf1113"). InnerVolumeSpecName "kube-api-access-rs52q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:48.199321 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:48.199248 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs52q\" (UniqueName: \"kubernetes.io/projected/6dee7e19-6aeb-46a7-b763-52cbd3cf1113-kube-api-access-rs52q\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:48.199321 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:48.199274 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dee7e19-6aeb-46a7-b763-52cbd3cf1113-plugins-config-volume\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:48.850738 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:48.850666 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-04679c-86c54d7f84-prhpr" Oct 30 20:14:48.850738 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:48.850706 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-04679c-86c54d7f84-prhpr" event={"ID":"191cdc99-91e9-49de-9006-66e40aa246b6","Type":"ContainerDied","Data":"0efef310c6440fc5e545adf93688a30d4efa9e64aadc06eb544eedd5d152c078"} Oct 30 20:14:48.851210 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:48.850763 4292 scope.go:122] "RemoveContainer" containerID="96dd42c675b063c1f91ab2385392830cdaf9832d9e83ebefc10981e5640605df" Oct 30 20:14:48.852257 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:48.852230 4292 generic.go:356] "Generic (PLEG): container finished" podID="6dee7e19-6aeb-46a7-b763-52cbd3cf1113" containerID="3dbf1983887a6ed07df40468dbfd9096f7b13f2b7dba14318e7a86f03a6a7025" exitCode=0 Oct 30 20:14:48.852371 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:48.852270 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-04679c-696cc7bfd7-zdpv7" Oct 30 20:14:48.852371 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:48.852321 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-04679c-696cc7bfd7-zdpv7" event={"ID":"6dee7e19-6aeb-46a7-b763-52cbd3cf1113","Type":"ContainerDied","Data":"3dbf1983887a6ed07df40468dbfd9096f7b13f2b7dba14318e7a86f03a6a7025"} Oct 30 20:14:48.852484 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:48.852379 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-04679c-696cc7bfd7-zdpv7" event={"ID":"6dee7e19-6aeb-46a7-b763-52cbd3cf1113","Type":"ContainerDied","Data":"97212bdc0cbce7b389dfb5093eb79a621e52888367481bc54261d9bb831946d2"} Oct 30 20:14:48.861776 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:48.861752 4292 scope.go:122] "RemoveContainer" containerID="3dbf1983887a6ed07df40468dbfd9096f7b13f2b7dba14318e7a86f03a6a7025" Oct 30 20:14:48.870501 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:48.870483 4292 scope.go:122] "RemoveContainer" containerID="3dbf1983887a6ed07df40468dbfd9096f7b13f2b7dba14318e7a86f03a6a7025" Oct 30 20:14:48.870937 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:14:48.870912 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dbf1983887a6ed07df40468dbfd9096f7b13f2b7dba14318e7a86f03a6a7025\": container with ID starting with 3dbf1983887a6ed07df40468dbfd9096f7b13f2b7dba14318e7a86f03a6a7025 not found: ID does not exist" containerID="3dbf1983887a6ed07df40468dbfd9096f7b13f2b7dba14318e7a86f03a6a7025" Oct 30 20:14:48.871015 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:48.870947 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dbf1983887a6ed07df40468dbfd9096f7b13f2b7dba14318e7a86f03a6a7025"} err="failed to get container status \"3dbf1983887a6ed07df40468dbfd9096f7b13f2b7dba14318e7a86f03a6a7025\": rpc error: code = NotFound desc = could not find container \"3dbf1983887a6ed07df40468dbfd9096f7b13f2b7dba14318e7a86f03a6a7025\": container with ID starting with 3dbf1983887a6ed07df40468dbfd9096f7b13f2b7dba14318e7a86f03a6a7025 not found: ID does not exist" Oct 30 20:14:49.022008 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:49.021975 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-04679c-86c54d7f84-prhpr"] Oct 30 20:14:49.072603 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:49.072574 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-04679c-86c54d7f84-prhpr"] Oct 30 20:14:49.137098 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:49.137023 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-04679c-696cc7bfd7-zdpv7"] Oct 30 20:14:49.140793 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:49.140770 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-04679c-696cc7bfd7-zdpv7"] Oct 30 20:14:50.581878 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:50.581848 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="191cdc99-91e9-49de-9006-66e40aa246b6" path="/var/lib/kubelet/pods/191cdc99-91e9-49de-9006-66e40aa246b6/volumes" Oct 30 20:14:50.582272 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:50.582227 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dee7e19-6aeb-46a7-b763-52cbd3cf1113" path="/var/lib/kubelet/pods/6dee7e19-6aeb-46a7-b763-52cbd3cf1113/volumes" Oct 30 20:14:53.121629 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:53.121595 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66699386-qwvrm"] Oct 30 20:14:53.122043 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:53.122028 4292 cpu_manager.go:414] "containerMap: removing container" podUID="6dee7e19-6aeb-46a7-b763-52cbd3cf1113" containerName="payload-pre-processing" Oct 30 20:14:53.122099 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:53.122046 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dee7e19-6aeb-46a7-b763-52cbd3cf1113" containerName="payload-pre-processing" Oct 30 20:14:53.122099 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:53.122059 4292 cpu_manager.go:414] "containerMap: removing container" podUID="432c0a5c-9c82-4d91-abe6-189b090f17d9" containerName="revoke-keys" Oct 30 20:14:53.122099 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:53.122067 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="432c0a5c-9c82-4d91-abe6-189b090f17d9" containerName="revoke-keys" Oct 30 20:14:53.122207 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:53.122104 4292 cpu_manager.go:414] "containerMap: removing container" podUID="191cdc99-91e9-49de-9006-66e40aa246b6" containerName="payload-processing" Oct 30 20:14:53.122207 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:53.122111 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="191cdc99-91e9-49de-9006-66e40aa246b6" containerName="payload-processing" Oct 30 20:14:53.122207 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:53.122191 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dee7e19-6aeb-46a7-b763-52cbd3cf1113" containerName="payload-pre-processing" Oct 30 20:14:53.122207 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:53.122205 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="432c0a5c-9c82-4d91-abe6-189b090f17d9" containerName="revoke-keys" Oct 30 20:14:53.122321 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:53.122214 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="191cdc99-91e9-49de-9006-66e40aa246b6" containerName="payload-processing" Oct 30 20:14:53.127028 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:53.127008 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66699386-qwvrm" Oct 30 20:14:53.181708 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:53.181674 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66699386-qwvrm"] Oct 30 20:14:53.228136 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:53.228112 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:14:53.228276 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:53.228152 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\"" Oct 30 20:14:53.252440 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:53.252411 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz79x\" (UniqueName: \"kubernetes.io/projected/17724efe-9918-459d-ac49-789619f7da5f-kube-api-access-hz79x\") pod \"maas-api-revoke-sub-66699386-qwvrm\" (UID: \"17724efe-9918-459d-ac49-789619f7da5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66699386-qwvrm" Oct 30 20:14:53.252590 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:53.252454 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17724efe-9918-459d-ac49-789619f7da5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66699386-qwvrm\" (UID: \"17724efe-9918-459d-ac49-789619f7da5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66699386-qwvrm" Oct 30 20:14:53.252689 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:53.252670 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17724efe-9918-459d-ac49-789619f7da5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66699386-qwvrm\" (UID: \"17724efe-9918-459d-ac49-789619f7da5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66699386-qwvrm" Oct 30 20:14:53.354061 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:53.354034 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17724efe-9918-459d-ac49-789619f7da5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66699386-qwvrm\" (UID: \"17724efe-9918-459d-ac49-789619f7da5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66699386-qwvrm" Oct 30 20:14:53.354231 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:53.354067 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz79x\" (UniqueName: \"kubernetes.io/projected/17724efe-9918-459d-ac49-789619f7da5f-kube-api-access-hz79x\") pod \"maas-api-revoke-sub-66699386-qwvrm\" (UID: \"17724efe-9918-459d-ac49-789619f7da5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66699386-qwvrm" Oct 30 20:14:53.354231 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:53.354087 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17724efe-9918-459d-ac49-789619f7da5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66699386-qwvrm\" (UID: \"17724efe-9918-459d-ac49-789619f7da5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66699386-qwvrm" Oct 30 20:14:53.354789 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:53.354765 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17724efe-9918-459d-ac49-789619f7da5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66699386-qwvrm\" (UID: \"17724efe-9918-459d-ac49-789619f7da5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66699386-qwvrm" Oct 30 20:14:53.356416 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:53.356391 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17724efe-9918-459d-ac49-789619f7da5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66699386-qwvrm\" (UID: \"17724efe-9918-459d-ac49-789619f7da5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66699386-qwvrm" Oct 30 20:14:53.416957 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:53.416893 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz79x\" (UniqueName: \"kubernetes.io/projected/17724efe-9918-459d-ac49-789619f7da5f-kube-api-access-hz79x\") pod \"maas-api-revoke-sub-66699386-qwvrm\" (UID: \"17724efe-9918-459d-ac49-789619f7da5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66699386-qwvrm" Oct 30 20:14:53.439635 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:53.439610 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66699386-qwvrm" Oct 30 20:14:53.816677 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:53.816646 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66699386-qwvrm"] Oct 30 20:14:53.818265 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:14:53.818238 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17724efe_9918_459d_ac49_789619f7da5f.slice/crio-0097e83e11fad6e66cd47cca561ee25716ea2dc51f3c78ea13ec7767abed556d WatchSource:0}: Error finding container 0097e83e11fad6e66cd47cca561ee25716ea2dc51f3c78ea13ec7767abed556d: Status 404 returned error can't find the container with id 0097e83e11fad6e66cd47cca561ee25716ea2dc51f3c78ea13ec7767abed556d Oct 30 20:14:53.891055 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:53.891027 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66699386-qwvrm" event={"ID":"17724efe-9918-459d-ac49-789619f7da5f","Type":"ContainerStarted","Data":"0097e83e11fad6e66cd47cca561ee25716ea2dc51f3c78ea13ec7767abed556d"} Oct 30 20:14:54.898454 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:54.898420 4292 generic.go:356] "Generic (PLEG): container finished" podID="17724efe-9918-459d-ac49-789619f7da5f" containerID="6f17ac3fa52c5b1e1ca129ff676da9f4d892dd948566c8f81798be28e59a2c09" exitCode=0 Oct 30 20:14:54.898847 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:54.898509 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66699386-qwvrm" event={"ID":"17724efe-9918-459d-ac49-789619f7da5f","Type":"ContainerDied","Data":"6f17ac3fa52c5b1e1ca129ff676da9f4d892dd948566c8f81798be28e59a2c09"} Oct 30 20:14:56.034720 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:56.034688 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66699386-qwvrm" Oct 30 20:14:56.083092 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:56.082017 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17724efe-9918-459d-ac49-789619f7da5f-kube-api-access-hz79x\" (UniqueName: \"kubernetes.io/projected/17724efe-9918-459d-ac49-789619f7da5f-kube-api-access-hz79x\") pod \"17724efe-9918-459d-ac49-789619f7da5f\" (UID: \"17724efe-9918-459d-ac49-789619f7da5f\") " Oct 30 20:14:56.083092 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:56.082071 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17724efe-9918-459d-ac49-789619f7da5f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17724efe-9918-459d-ac49-789619f7da5f-maas-api-xks-ca\") pod \"17724efe-9918-459d-ac49-789619f7da5f\" (UID: \"17724efe-9918-459d-ac49-789619f7da5f\") " Oct 30 20:14:56.083092 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:56.082157 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17724efe-9918-459d-ac49-789619f7da5f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17724efe-9918-459d-ac49-789619f7da5f-maas-api-openshift-ca\") pod \"17724efe-9918-459d-ac49-789619f7da5f\" (UID: \"17724efe-9918-459d-ac49-789619f7da5f\") " Oct 30 20:14:56.083855 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:56.083558 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17724efe-9918-459d-ac49-789619f7da5f-maas-api-openshift-ca" pod "17724efe-9918-459d-ac49-789619f7da5f" (UID: "17724efe-9918-459d-ac49-789619f7da5f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:14:56.091546 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:56.091517 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17724efe-9918-459d-ac49-789619f7da5f-kube-api-access-hz79x" pod "17724efe-9918-459d-ac49-789619f7da5f" (UID: "17724efe-9918-459d-ac49-789619f7da5f"). InnerVolumeSpecName "kube-api-access-hz79x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:56.095386 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:56.095336 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17724efe-9918-459d-ac49-789619f7da5f-maas-api-xks-ca" pod "17724efe-9918-459d-ac49-789619f7da5f" (UID: "17724efe-9918-459d-ac49-789619f7da5f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:14:56.183944 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:56.183868 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz79x\" (UniqueName: \"kubernetes.io/projected/17724efe-9918-459d-ac49-789619f7da5f-kube-api-access-hz79x\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:56.183944 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:56.183895 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17724efe-9918-459d-ac49-789619f7da5f-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:56.183944 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:56.183906 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17724efe-9918-459d-ac49-789619f7da5f-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:56.910069 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:56.909994 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66699386-qwvrm" Oct 30 20:14:56.910069 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:56.910030 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66699386-qwvrm" event={"ID":"17724efe-9918-459d-ac49-789619f7da5f","Type":"ContainerDied","Data":"0097e83e11fad6e66cd47cca561ee25716ea2dc51f3c78ea13ec7767abed556d"} Oct 30 20:14:56.910291 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:14:56.910077 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0097e83e11fad6e66cd47cca561ee25716ea2dc51f3c78ea13ec7767abed556d" Oct 30 20:15:00.162992 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:00.162955 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889855-s4n4l"] Oct 30 20:15:00.163617 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:00.163597 4292 cpu_manager.go:414] "containerMap: removing container" podUID="17724efe-9918-459d-ac49-789619f7da5f" containerName="revoke-keys" Oct 30 20:15:00.163704 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:00.163619 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17724efe-9918-459d-ac49-789619f7da5f" containerName="revoke-keys" Oct 30 20:15:00.163784 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:00.163772 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="17724efe-9918-459d-ac49-789619f7da5f" containerName="revoke-keys" Oct 30 20:15:00.166886 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:00.166865 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-389c4e-29889855-lbsk6"] Oct 30 20:15:00.167003 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:00.166976 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889855-s4n4l" Oct 30 20:15:00.170972 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:00.170953 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d5d3ba-29889855-bd72m"] Oct 30 20:15:00.171092 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:00.171073 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-389c4e-29889855-lbsk6" Oct 30 20:15:00.174212 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:00.174190 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d5d3ba-29889855-bd72m" Oct 30 20:15:00.213535 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:00.213503 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889855-j5krt"] Oct 30 20:15:00.216994 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:00.216978 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889855-j5krt" Oct 30 20:15:00.222374 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:00.222328 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\"" Oct 30 20:15:00.226275 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:00.226253 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h445p\" (UniqueName: \"kubernetes.io/projected/219a0a4f-88b9-4f4c-92c6-4a8ffa788842-kube-api-access-h445p\") pod \"maas-api-key-cleanup-e2e-praxis-a-389c4e-29889855-lbsk6\" (UID: \"219a0a4f-88b9-4f4c-92c6-4a8ffa788842\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-389c4e-29889855-lbsk6" Oct 30 20:15:00.226374 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:00.226297 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a4c41bd-fc28-4552-be12-170b75585dae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-d5d3ba-29889855-bd72m\" (UID: \"0a4c41bd-fc28-4552-be12-170b75585dae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d5d3ba-29889855-bd72m" Oct 30 20:15:00.226431 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:00.226384 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/219a0a4f-88b9-4f4c-92c6-4a8ffa788842-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-389c4e-29889855-lbsk6\" (UID: \"219a0a4f-88b9-4f4c-92c6-4a8ffa788842\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-389c4e-29889855-lbsk6" Oct 30 20:15:00.226431 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:00.226404 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e455047b-0948-4538-a504-9e0a3324b00c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889855-s4n4l\" (UID: \"e455047b-0948-4538-a504-9e0a3324b00c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889855-s4n4l" Oct 30 20:15:00.226594 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:00.226571 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2zqf\" (UniqueName: \"kubernetes.io/projected/e455047b-0948-4538-a504-9e0a3324b00c-kube-api-access-d2zqf\") pod \"maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889855-s4n4l\" (UID: \"e455047b-0948-4538-a504-9e0a3324b00c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889855-s4n4l" Oct 30 20:15:00.226650 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:00.226633 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgsln\" (UniqueName: \"kubernetes.io/projected/0a4c41bd-fc28-4552-be12-170b75585dae-kube-api-access-jgsln\") pod \"maas-api-key-cleanup-e2e-praxis-a-d5d3ba-29889855-bd72m\" (UID: \"0a4c41bd-fc28-4552-be12-170b75585dae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d5d3ba-29889855-bd72m" Oct 30 20:15:00.261042 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:00.261017 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-389c4e-29889855-lbsk6"] Oct 30 20:15:00.264315 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:00.264291 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889855-s4n4l"] Oct 30 20:15:00.266776 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:00.266759 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d5d3ba-29889855-bd72m"] Oct 30 20:15:00.327840 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:00.327810 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889855-j5krt"] Oct 30 20:15:00.328105 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:00.328081 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2zqf\" (UniqueName: \"kubernetes.io/projected/e455047b-0948-4538-a504-9e0a3324b00c-kube-api-access-d2zqf\") pod \"maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889855-s4n4l\" (UID: \"e455047b-0948-4538-a504-9e0a3324b00c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889855-s4n4l" Oct 30 20:15:00.328170 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:00.328126 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgsln\" (UniqueName: \"kubernetes.io/projected/0a4c41bd-fc28-4552-be12-170b75585dae-kube-api-access-jgsln\") pod \"maas-api-key-cleanup-e2e-praxis-a-d5d3ba-29889855-bd72m\" (UID: \"0a4c41bd-fc28-4552-be12-170b75585dae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d5d3ba-29889855-bd72m" Oct 30 20:15:00.328219 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:00.328168 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c74471c-9ea5-486f-b083-efee6a92777d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-4ec609-29889855-j5krt\" (UID: \"6c74471c-9ea5-486f-b083-efee6a92777d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889855-j5krt" Oct 30 20:15:00.328219 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:00.328190 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h445p\" (UniqueName: \"kubernetes.io/projected/219a0a4f-88b9-4f4c-92c6-4a8ffa788842-kube-api-access-h445p\") pod \"maas-api-key-cleanup-e2e-praxis-a-389c4e-29889855-lbsk6\" (UID: \"219a0a4f-88b9-4f4c-92c6-4a8ffa788842\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-389c4e-29889855-lbsk6" Oct 30 20:15:00.328219 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:00.328214 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfdkz\" (UniqueName: \"kubernetes.io/projected/6c74471c-9ea5-486f-b083-efee6a92777d-kube-api-access-zfdkz\") pod \"maas-api-key-cleanup-e2e-worker-w4-4ec609-29889855-j5krt\" (UID: \"6c74471c-9ea5-486f-b083-efee6a92777d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889855-j5krt" Oct 30 20:15:00.328343 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:00.328309 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a4c41bd-fc28-4552-be12-170b75585dae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-d5d3ba-29889855-bd72m\" (UID: \"0a4c41bd-fc28-4552-be12-170b75585dae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d5d3ba-29889855-bd72m" Oct 30 20:15:00.328432 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:00.328389 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/219a0a4f-88b9-4f4c-92c6-4a8ffa788842-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-389c4e-29889855-lbsk6\" (UID: \"219a0a4f-88b9-4f4c-92c6-4a8ffa788842\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-389c4e-29889855-lbsk6" Oct 30 20:15:00.328432 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:00.328425 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e455047b-0948-4538-a504-9e0a3324b00c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889855-s4n4l\" (UID: \"e455047b-0948-4538-a504-9e0a3324b00c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889855-s4n4l" Oct 30 20:15:00.328961 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:00.328937 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a4c41bd-fc28-4552-be12-170b75585dae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-d5d3ba-29889855-bd72m\" (UID: \"0a4c41bd-fc28-4552-be12-170b75585dae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d5d3ba-29889855-bd72m" Oct 30 20:15:00.329149 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:00.329129 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/219a0a4f-88b9-4f4c-92c6-4a8ffa788842-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-389c4e-29889855-lbsk6\" (UID: \"219a0a4f-88b9-4f4c-92c6-4a8ffa788842\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-389c4e-29889855-lbsk6" Oct 30 20:15:00.329204 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:00.329183 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e455047b-0948-4538-a504-9e0a3324b00c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889855-s4n4l\" (UID: \"e455047b-0948-4538-a504-9e0a3324b00c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889855-s4n4l" Oct 30 20:15:00.392482 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:00.392446 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h445p\" (UniqueName: \"kubernetes.io/projected/219a0a4f-88b9-4f4c-92c6-4a8ffa788842-kube-api-access-h445p\") pod \"maas-api-key-cleanup-e2e-praxis-a-389c4e-29889855-lbsk6\" (UID: \"219a0a4f-88b9-4f4c-92c6-4a8ffa788842\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-389c4e-29889855-lbsk6" Oct 30 20:15:00.392619 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:00.392541 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgsln\" (UniqueName: \"kubernetes.io/projected/0a4c41bd-fc28-4552-be12-170b75585dae-kube-api-access-jgsln\") pod \"maas-api-key-cleanup-e2e-praxis-a-d5d3ba-29889855-bd72m\" (UID: \"0a4c41bd-fc28-4552-be12-170b75585dae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d5d3ba-29889855-bd72m" Oct 30 20:15:00.392619 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:00.392541 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2zqf\" (UniqueName: \"kubernetes.io/projected/e455047b-0948-4538-a504-9e0a3324b00c-kube-api-access-d2zqf\") pod \"maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889855-s4n4l\" (UID: \"e455047b-0948-4538-a504-9e0a3324b00c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889855-s4n4l" Oct 30 20:15:00.429667 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:00.429602 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c74471c-9ea5-486f-b083-efee6a92777d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-4ec609-29889855-j5krt\" (UID: \"6c74471c-9ea5-486f-b083-efee6a92777d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889855-j5krt" Oct 30 20:15:00.429667 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:00.429641 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfdkz\" (UniqueName: \"kubernetes.io/projected/6c74471c-9ea5-486f-b083-efee6a92777d-kube-api-access-zfdkz\") pod \"maas-api-key-cleanup-e2e-worker-w4-4ec609-29889855-j5krt\" (UID: \"6c74471c-9ea5-486f-b083-efee6a92777d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889855-j5krt" Oct 30 20:15:00.430232 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:00.430205 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c74471c-9ea5-486f-b083-efee6a92777d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-4ec609-29889855-j5krt\" (UID: \"6c74471c-9ea5-486f-b083-efee6a92777d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889855-j5krt" Oct 30 20:15:00.481155 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:00.481128 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889855-s4n4l" Oct 30 20:15:00.488363 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:00.488333 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-389c4e-29889855-lbsk6" Oct 30 20:15:00.494987 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:00.494958 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d5d3ba-29889855-bd72m" Oct 30 20:15:00.497400 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:00.497251 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfdkz\" (UniqueName: \"kubernetes.io/projected/6c74471c-9ea5-486f-b083-efee6a92777d-kube-api-access-zfdkz\") pod \"maas-api-key-cleanup-e2e-worker-w4-4ec609-29889855-j5krt\" (UID: \"6c74471c-9ea5-486f-b083-efee6a92777d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889855-j5krt" Oct 30 20:15:00.529595 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:00.529568 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889855-j5krt" Oct 30 20:15:00.866253 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:15:00.866220 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode455047b_0948_4538_a504_9e0a3324b00c.slice/crio-02f76cf77a781c0918307a9e284ea20ddd7701c1dcd42ccca29077f6ba429819 WatchSource:0}: Error finding container 02f76cf77a781c0918307a9e284ea20ddd7701c1dcd42ccca29077f6ba429819: Status 404 returned error can't find the container with id 02f76cf77a781c0918307a9e284ea20ddd7701c1dcd42ccca29077f6ba429819 Oct 30 20:15:00.870592 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:00.870568 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889855-s4n4l"] Oct 30 20:15:00.910931 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:15:00.910874 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a4c41bd_fc28_4552_be12_170b75585dae.slice/crio-03254b53705263eb8d4e89de2e51b21e31b8b539c749186681c01f150f37c9db WatchSource:0}: Error finding container 03254b53705263eb8d4e89de2e51b21e31b8b539c749186681c01f150f37c9db: Status 404 returned error can't find the container with id 03254b53705263eb8d4e89de2e51b21e31b8b539c749186681c01f150f37c9db Oct 30 20:15:00.915167 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:00.915141 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d5d3ba-29889855-bd72m"] Oct 30 20:15:00.915423 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:15:00.915396 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod219a0a4f_88b9_4f4c_92c6_4a8ffa788842.slice/crio-96c00913f076c959f21beddb714c5b6d12773415b0b41bf24b948d6036602840 WatchSource:0}: Error finding container 96c00913f076c959f21beddb714c5b6d12773415b0b41bf24b948d6036602840: Status 404 returned error can't find the container with id 96c00913f076c959f21beddb714c5b6d12773415b0b41bf24b948d6036602840 Oct 30 20:15:00.918252 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:00.918233 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-389c4e-29889855-lbsk6"] Oct 30 20:15:00.936537 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:00.936514 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d5d3ba-29889855-bd72m" event={"ID":"0a4c41bd-fc28-4552-be12-170b75585dae","Type":"ContainerStarted","Data":"03254b53705263eb8d4e89de2e51b21e31b8b539c749186681c01f150f37c9db"} Oct 30 20:15:00.937807 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:00.937782 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889855-s4n4l" event={"ID":"e455047b-0948-4538-a504-9e0a3324b00c","Type":"ContainerStarted","Data":"02f76cf77a781c0918307a9e284ea20ddd7701c1dcd42ccca29077f6ba429819"} Oct 30 20:15:00.938881 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:00.938862 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-389c4e-29889855-lbsk6" event={"ID":"219a0a4f-88b9-4f4c-92c6-4a8ffa788842","Type":"ContainerStarted","Data":"96c00913f076c959f21beddb714c5b6d12773415b0b41bf24b948d6036602840"} Oct 30 20:15:00.975514 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:15:00.975489 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c74471c_9ea5_486f_b083_efee6a92777d.slice/crio-72835a2ddaeb42d5561c1a9f7fbab3ba002e845666f3b4c3d23a2952e0ddf736 WatchSource:0}: Error finding container 72835a2ddaeb42d5561c1a9f7fbab3ba002e845666f3b4c3d23a2952e0ddf736: Status 404 returned error can't find the container with id 72835a2ddaeb42d5561c1a9f7fbab3ba002e845666f3b4c3d23a2952e0ddf736 Oct 30 20:15:00.979217 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:00.979196 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889855-j5krt"] Oct 30 20:15:01.302140 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:01.302104 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9c6d29b-htqdr"] Oct 30 20:15:01.305968 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:01.305948 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c6d29b-htqdr" Oct 30 20:15:01.359604 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:01.359565 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:15:01.360076 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:01.360052 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9c6d29b-htqdr"] Oct 30 20:15:01.447823 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:01.447791 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlxfb\" (UniqueName: \"kubernetes.io/projected/c4374031-874c-47dd-b5a0-4bbd49dde2ba-kube-api-access-mlxfb\") pod \"maas-api-revoke-sub-d9c6d29b-htqdr\" (UID: \"c4374031-874c-47dd-b5a0-4bbd49dde2ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c6d29b-htqdr" Oct 30 20:15:01.447983 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:01.447834 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4374031-874c-47dd-b5a0-4bbd49dde2ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9c6d29b-htqdr\" (UID: \"c4374031-874c-47dd-b5a0-4bbd49dde2ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c6d29b-htqdr" Oct 30 20:15:01.447983 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:01.447858 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4374031-874c-47dd-b5a0-4bbd49dde2ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9c6d29b-htqdr\" (UID: \"c4374031-874c-47dd-b5a0-4bbd49dde2ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c6d29b-htqdr" Oct 30 20:15:01.556926 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:01.556890 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlxfb\" (UniqueName: \"kubernetes.io/projected/c4374031-874c-47dd-b5a0-4bbd49dde2ba-kube-api-access-mlxfb\") pod \"maas-api-revoke-sub-d9c6d29b-htqdr\" (UID: \"c4374031-874c-47dd-b5a0-4bbd49dde2ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c6d29b-htqdr" Oct 30 20:15:01.557146 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:01.556957 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4374031-874c-47dd-b5a0-4bbd49dde2ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9c6d29b-htqdr\" (UID: \"c4374031-874c-47dd-b5a0-4bbd49dde2ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c6d29b-htqdr" Oct 30 20:15:01.557146 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:01.557014 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4374031-874c-47dd-b5a0-4bbd49dde2ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9c6d29b-htqdr\" (UID: \"c4374031-874c-47dd-b5a0-4bbd49dde2ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c6d29b-htqdr" Oct 30 20:15:01.558141 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:01.558109 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4374031-874c-47dd-b5a0-4bbd49dde2ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9c6d29b-htqdr\" (UID: \"c4374031-874c-47dd-b5a0-4bbd49dde2ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c6d29b-htqdr" Oct 30 20:15:01.560940 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:01.560915 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4374031-874c-47dd-b5a0-4bbd49dde2ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9c6d29b-htqdr\" (UID: \"c4374031-874c-47dd-b5a0-4bbd49dde2ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c6d29b-htqdr" Oct 30 20:15:01.621824 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:01.621794 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlxfb\" (UniqueName: \"kubernetes.io/projected/c4374031-874c-47dd-b5a0-4bbd49dde2ba-kube-api-access-mlxfb\") pod \"maas-api-revoke-sub-d9c6d29b-htqdr\" (UID: \"c4374031-874c-47dd-b5a0-4bbd49dde2ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c6d29b-htqdr" Oct 30 20:15:01.919326 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:01.919299 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c6d29b-htqdr" Oct 30 20:15:01.946641 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:01.946615 4292 generic.go:356] "Generic (PLEG): container finished" podID="e455047b-0948-4538-a504-9e0a3324b00c" containerID="c966f9cf743d5914499fbc8b56e8f46c9532d37a8cca022ca88f876cca408ce5" exitCode=60 Oct 30 20:15:01.946818 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:01.946705 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889855-s4n4l" event={"ID":"e455047b-0948-4538-a504-9e0a3324b00c","Type":"ContainerDied","Data":"c966f9cf743d5914499fbc8b56e8f46c9532d37a8cca022ca88f876cca408ce5"} Oct 30 20:15:01.946969 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:01.946951 4292 scope.go:122] "RemoveContainer" containerID="c966f9cf743d5914499fbc8b56e8f46c9532d37a8cca022ca88f876cca408ce5" Oct 30 20:15:01.948589 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:01.948564 4292 generic.go:356] "Generic (PLEG): container finished" podID="219a0a4f-88b9-4f4c-92c6-4a8ffa788842" containerID="bd9e3b3ce8412c69792c758d0557adc7494d06730062cb8f6082c8498200d836" exitCode=60 Oct 30 20:15:01.948700 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:01.948650 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-389c4e-29889855-lbsk6" event={"ID":"219a0a4f-88b9-4f4c-92c6-4a8ffa788842","Type":"ContainerDied","Data":"bd9e3b3ce8412c69792c758d0557adc7494d06730062cb8f6082c8498200d836"} Oct 30 20:15:01.948904 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:01.948885 4292 scope.go:122] "RemoveContainer" containerID="bd9e3b3ce8412c69792c758d0557adc7494d06730062cb8f6082c8498200d836" Oct 30 20:15:01.953822 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:01.951385 4292 generic.go:356] "Generic (PLEG): container finished" podID="6c74471c-9ea5-486f-b083-efee6a92777d" containerID="37d47786eba1c955d791c2b2a4d1c7fd12f14da9d3add3f7e9f78af3c83840d2" exitCode=60 Oct 30 20:15:01.953822 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:01.951414 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889855-j5krt" event={"ID":"6c74471c-9ea5-486f-b083-efee6a92777d","Type":"ContainerDied","Data":"37d47786eba1c955d791c2b2a4d1c7fd12f14da9d3add3f7e9f78af3c83840d2"} Oct 30 20:15:01.953822 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:01.951480 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889855-j5krt" event={"ID":"6c74471c-9ea5-486f-b083-efee6a92777d","Type":"ContainerStarted","Data":"72835a2ddaeb42d5561c1a9f7fbab3ba002e845666f3b4c3d23a2952e0ddf736"} Oct 30 20:15:01.953822 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:01.951700 4292 scope.go:122] "RemoveContainer" containerID="37d47786eba1c955d791c2b2a4d1c7fd12f14da9d3add3f7e9f78af3c83840d2" Oct 30 20:15:01.953822 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:01.953553 4292 generic.go:356] "Generic (PLEG): container finished" podID="0a4c41bd-fc28-4552-be12-170b75585dae" containerID="f5cf6e447feae5ade188588c674c2fd37785872c148dd22c1d01e0127dfbcf13" exitCode=7 Oct 30 20:15:01.953822 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:01.953618 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d5d3ba-29889855-bd72m" event={"ID":"0a4c41bd-fc28-4552-be12-170b75585dae","Type":"ContainerDied","Data":"f5cf6e447feae5ade188588c674c2fd37785872c148dd22c1d01e0127dfbcf13"} Oct 30 20:15:01.953822 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:01.953752 4292 scope.go:122] "RemoveContainer" containerID="f5cf6e447feae5ade188588c674c2fd37785872c148dd22c1d01e0127dfbcf13" Oct 30 20:15:02.257083 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:02.257057 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht"] Oct 30 20:15:02.261157 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:02.261131 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht" Oct 30 20:15:02.290706 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:15:02.290683 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4374031_874c_47dd_b5a0_4bbd49dde2ba.slice/crio-67cf008bb7683899e421717becd40bbc11f42284013cb64cec6c54488e189ba7 WatchSource:0}: Error finding container 67cf008bb7683899e421717becd40bbc11f42284013cb64cec6c54488e189ba7: Status 404 returned error can't find the container with id 67cf008bb7683899e421717becd40bbc11f42284013cb64cec6c54488e189ba7 Oct 30 20:15:02.295221 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:02.295197 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9c6d29b-htqdr"] Oct 30 20:15:02.320246 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:02.320217 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht"] Oct 30 20:15:02.320811 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:02.320794 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-98a19b-openshift-default-dockercfg-rhphv\"" Oct 30 20:15:02.365063 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:02.365025 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f9dd92b-3ba4-4d87-8162-df97a30a48d0-istio-podinfo\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht\" (UID: \"8f9dd92b-3ba4-4d87-8162-df97a30a48d0\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht" Oct 30 20:15:02.365206 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:02.365154 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f9dd92b-3ba4-4d87-8162-df97a30a48d0-istio-envoy\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht\" (UID: \"8f9dd92b-3ba4-4d87-8162-df97a30a48d0\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht" Oct 30 20:15:02.365206 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:02.365192 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f9dd92b-3ba4-4d87-8162-df97a30a48d0-istio-data\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht\" (UID: \"8f9dd92b-3ba4-4d87-8162-df97a30a48d0\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht" Oct 30 20:15:02.365302 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:02.365212 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f9dd92b-3ba4-4d87-8162-df97a30a48d0-credential-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht\" (UID: \"8f9dd92b-3ba4-4d87-8162-df97a30a48d0\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht" Oct 30 20:15:02.365302 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:02.365237 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f9dd92b-3ba4-4d87-8162-df97a30a48d0-istiod-ca-cert\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht\" (UID: \"8f9dd92b-3ba4-4d87-8162-df97a30a48d0\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht" Oct 30 20:15:02.365418 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:02.365388 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f9dd92b-3ba4-4d87-8162-df97a30a48d0-workload-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht\" (UID: \"8f9dd92b-3ba4-4d87-8162-df97a30a48d0\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht" Oct 30 20:15:02.365468 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:02.365415 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f9dd92b-3ba4-4d87-8162-df97a30a48d0-istio-token\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht\" (UID: \"8f9dd92b-3ba4-4d87-8162-df97a30a48d0\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht" Oct 30 20:15:02.365468 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:02.365439 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6ftt\" (UniqueName: \"kubernetes.io/projected/8f9dd92b-3ba4-4d87-8162-df97a30a48d0-kube-api-access-q6ftt\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht\" (UID: \"8f9dd92b-3ba4-4d87-8162-df97a30a48d0\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht" Oct 30 20:15:02.365558 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:02.365542 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f9dd92b-3ba4-4d87-8162-df97a30a48d0-workload-certs\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht\" (UID: \"8f9dd92b-3ba4-4d87-8162-df97a30a48d0\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht" Oct 30 20:15:02.466689 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:02.466660 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f9dd92b-3ba4-4d87-8162-df97a30a48d0-workload-certs\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht\" (UID: \"8f9dd92b-3ba4-4d87-8162-df97a30a48d0\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht" Oct 30 20:15:02.466790 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:02.466710 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f9dd92b-3ba4-4d87-8162-df97a30a48d0-istio-podinfo\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht\" (UID: \"8f9dd92b-3ba4-4d87-8162-df97a30a48d0\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht" Oct 30 20:15:02.466790 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:02.466784 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f9dd92b-3ba4-4d87-8162-df97a30a48d0-istio-envoy\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht\" (UID: \"8f9dd92b-3ba4-4d87-8162-df97a30a48d0\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht" Oct 30 20:15:02.466893 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:02.466818 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f9dd92b-3ba4-4d87-8162-df97a30a48d0-istio-data\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht\" (UID: \"8f9dd92b-3ba4-4d87-8162-df97a30a48d0\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht" Oct 30 20:15:02.466893 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:02.466845 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f9dd92b-3ba4-4d87-8162-df97a30a48d0-credential-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht\" (UID: \"8f9dd92b-3ba4-4d87-8162-df97a30a48d0\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht" Oct 30 20:15:02.467445 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:02.467052 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f9dd92b-3ba4-4d87-8162-df97a30a48d0-istiod-ca-cert\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht\" (UID: \"8f9dd92b-3ba4-4d87-8162-df97a30a48d0\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht" Oct 30 20:15:02.467445 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:02.467079 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f9dd92b-3ba4-4d87-8162-df97a30a48d0-workload-certs\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht\" (UID: \"8f9dd92b-3ba4-4d87-8162-df97a30a48d0\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht" Oct 30 20:15:02.467445 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:02.467123 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f9dd92b-3ba4-4d87-8162-df97a30a48d0-workload-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht\" (UID: \"8f9dd92b-3ba4-4d87-8162-df97a30a48d0\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht" Oct 30 20:15:02.467445 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:02.467151 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f9dd92b-3ba4-4d87-8162-df97a30a48d0-credential-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht\" (UID: \"8f9dd92b-3ba4-4d87-8162-df97a30a48d0\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht" Oct 30 20:15:02.467445 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:02.467155 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f9dd92b-3ba4-4d87-8162-df97a30a48d0-istio-token\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht\" (UID: \"8f9dd92b-3ba4-4d87-8162-df97a30a48d0\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht" Oct 30 20:15:02.467445 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:02.467223 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6ftt\" (UniqueName: \"kubernetes.io/projected/8f9dd92b-3ba4-4d87-8162-df97a30a48d0-kube-api-access-q6ftt\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht\" (UID: \"8f9dd92b-3ba4-4d87-8162-df97a30a48d0\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht" Oct 30 20:15:02.467445 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:02.467243 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f9dd92b-3ba4-4d87-8162-df97a30a48d0-istio-data\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht\" (UID: \"8f9dd92b-3ba4-4d87-8162-df97a30a48d0\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht" Oct 30 20:15:02.467445 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:02.467416 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f9dd92b-3ba4-4d87-8162-df97a30a48d0-workload-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht\" (UID: \"8f9dd92b-3ba4-4d87-8162-df97a30a48d0\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht" Oct 30 20:15:02.467949 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:02.467846 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f9dd92b-3ba4-4d87-8162-df97a30a48d0-istiod-ca-cert\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht\" (UID: \"8f9dd92b-3ba4-4d87-8162-df97a30a48d0\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht" Oct 30 20:15:02.469052 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:02.469033 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f9dd92b-3ba4-4d87-8162-df97a30a48d0-istio-podinfo\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht\" (UID: \"8f9dd92b-3ba4-4d87-8162-df97a30a48d0\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht" Oct 30 20:15:02.469793 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:02.469776 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f9dd92b-3ba4-4d87-8162-df97a30a48d0-istio-envoy\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht\" (UID: \"8f9dd92b-3ba4-4d87-8162-df97a30a48d0\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht" Oct 30 20:15:02.529990 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:02.529963 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f9dd92b-3ba4-4d87-8162-df97a30a48d0-istio-token\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht\" (UID: \"8f9dd92b-3ba4-4d87-8162-df97a30a48d0\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht" Oct 30 20:15:02.530166 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:02.530148 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6ftt\" (UniqueName: \"kubernetes.io/projected/8f9dd92b-3ba4-4d87-8162-df97a30a48d0-kube-api-access-q6ftt\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht\" (UID: \"8f9dd92b-3ba4-4d87-8162-df97a30a48d0\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht" Oct 30 20:15:02.575975 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:02.575906 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht" Oct 30 20:15:02.953827 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:15:02.953795 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f9dd92b_3ba4_4d87_8162_df97a30a48d0.slice/crio-513e7ad811b4c198baa46c0390dd393155127179d2707603b70e1ed15f4fcdef WatchSource:0}: Error finding container 513e7ad811b4c198baa46c0390dd393155127179d2707603b70e1ed15f4fcdef: Status 404 returned error can't find the container with id 513e7ad811b4c198baa46c0390dd393155127179d2707603b70e1ed15f4fcdef Oct 30 20:15:02.956556 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:02.956521 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:15:02.956665 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:02.956604 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:15:02.956665 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:02.956652 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:15:02.956985 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:02.956966 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht"] Oct 30 20:15:02.963028 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:02.963005 4292 generic.go:356] "Generic (PLEG): container finished" podID="e455047b-0948-4538-a504-9e0a3324b00c" containerID="e99ae5bcd1c114b26a677d4846bb4eb3743108b19046574c05903820ebccd0e2" exitCode=60 Oct 30 20:15:02.963138 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:02.963091 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889855-s4n4l" event={"ID":"e455047b-0948-4538-a504-9e0a3324b00c","Type":"ContainerDied","Data":"e99ae5bcd1c114b26a677d4846bb4eb3743108b19046574c05903820ebccd0e2"} Oct 30 20:15:02.963201 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:02.963145 4292 scope.go:122] "RemoveContainer" containerID="c966f9cf743d5914499fbc8b56e8f46c9532d37a8cca022ca88f876cca408ce5" Oct 30 20:15:02.963490 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:02.963464 4292 scope.go:122] "RemoveContainer" containerID="e99ae5bcd1c114b26a677d4846bb4eb3743108b19046574c05903820ebccd0e2" Oct 30 20:15:02.963774 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:15:02.963750 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889855-s4n4l_odh-ai-gateway-infra(e455047b-0948-4538-a504-9e0a3324b00c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889855-s4n4l" podUID="e455047b-0948-4538-a504-9e0a3324b00c" Oct 30 20:15:02.964956 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:02.964932 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-389c4e-29889855-lbsk6" event={"ID":"219a0a4f-88b9-4f4c-92c6-4a8ffa788842","Type":"ContainerStarted","Data":"f6f8553d16b6bf4411be31a38d3f7f4e1b7cb37e49e6134030e109e8a43ddc80"} Oct 30 20:15:02.967173 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:02.967152 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889855-j5krt" event={"ID":"6c74471c-9ea5-486f-b083-efee6a92777d","Type":"ContainerStarted","Data":"f132942b5f544821944ddce0d13c5a1b9ca92c5389a84a6193cb187dc9cf8af9"} Oct 30 20:15:02.969305 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:02.969165 4292 generic.go:356] "Generic (PLEG): container finished" podID="0a4c41bd-fc28-4552-be12-170b75585dae" containerID="8b7acbe7c84d4db2358e3cfd9df5ac0e0d49a40fcf822f86ffad7acde9331dcc" exitCode=7 Oct 30 20:15:02.969305 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:02.969245 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d5d3ba-29889855-bd72m" event={"ID":"0a4c41bd-fc28-4552-be12-170b75585dae","Type":"ContainerDied","Data":"8b7acbe7c84d4db2358e3cfd9df5ac0e0d49a40fcf822f86ffad7acde9331dcc"} Oct 30 20:15:02.969645 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:02.969623 4292 scope.go:122] "RemoveContainer" containerID="8b7acbe7c84d4db2358e3cfd9df5ac0e0d49a40fcf822f86ffad7acde9331dcc" Oct 30 20:15:02.969994 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:15:02.969974 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-d5d3ba-29889855-bd72m_odh-ai-gateway-infra(0a4c41bd-fc28-4552-be12-170b75585dae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d5d3ba-29889855-bd72m" podUID="0a4c41bd-fc28-4552-be12-170b75585dae" Oct 30 20:15:02.971560 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:02.971523 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c6d29b-htqdr" event={"ID":"c4374031-874c-47dd-b5a0-4bbd49dde2ba","Type":"ContainerStarted","Data":"b97450241ed4ebcd7b191d65370b987fd11ff0b5826acb56bec88465fb625d1d"} Oct 30 20:15:02.971674 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:02.971567 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c6d29b-htqdr" event={"ID":"c4374031-874c-47dd-b5a0-4bbd49dde2ba","Type":"ContainerStarted","Data":"67cf008bb7683899e421717becd40bbc11f42284013cb64cec6c54488e189ba7"} Oct 30 20:15:02.978219 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:02.978200 4292 scope.go:122] "RemoveContainer" containerID="f5cf6e447feae5ade188588c674c2fd37785872c148dd22c1d01e0127dfbcf13" Oct 30 20:15:03.316939 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:03.316879 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-389c4e-29889855-lbsk6" podStartSLOduration=3.316860485 podStartE2EDuration="3.316860485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:03.316155007 +0000 UTC m=+3006.834470464" watchObservedRunningTime="2026-10-30 20:15:03.316860485 +0000 UTC m=+3006.835175941" Oct 30 20:15:03.436743 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:03.436691 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889855-j5krt" podStartSLOduration=3.436677235 podStartE2EDuration="3.436677235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:03.43623876 +0000 UTC m=+3006.954554234" watchObservedRunningTime="2026-10-30 20:15:03.436677235 +0000 UTC m=+3006.954992702" Oct 30 20:15:03.564375 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:03.563457 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c6d29b-htqdr" podStartSLOduration=2.563439527 podStartE2EDuration="2.563439527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:03.561750086 +0000 UTC m=+3007.080065555" watchObservedRunningTime="2026-10-30 20:15:03.563439527 +0000 UTC m=+3007.081754984" Oct 30 20:15:03.987111 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:03.987079 4292 scope.go:122] "RemoveContainer" containerID="e99ae5bcd1c114b26a677d4846bb4eb3743108b19046574c05903820ebccd0e2" Oct 30 20:15:03.987419 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:15:03.987393 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889855-s4n4l_odh-ai-gateway-infra(e455047b-0948-4538-a504-9e0a3324b00c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889855-s4n4l" podUID="e455047b-0948-4538-a504-9e0a3324b00c" Oct 30 20:15:03.988993 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:03.988967 4292 generic.go:356] "Generic (PLEG): container finished" podID="219a0a4f-88b9-4f4c-92c6-4a8ffa788842" containerID="f6f8553d16b6bf4411be31a38d3f7f4e1b7cb37e49e6134030e109e8a43ddc80" exitCode=60 Oct 30 20:15:03.989113 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:03.989041 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-389c4e-29889855-lbsk6" event={"ID":"219a0a4f-88b9-4f4c-92c6-4a8ffa788842","Type":"ContainerDied","Data":"f6f8553d16b6bf4411be31a38d3f7f4e1b7cb37e49e6134030e109e8a43ddc80"} Oct 30 20:15:03.989113 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:03.989104 4292 scope.go:122] "RemoveContainer" containerID="bd9e3b3ce8412c69792c758d0557adc7494d06730062cb8f6082c8498200d836" Oct 30 20:15:03.989267 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:03.989249 4292 scope.go:122] "RemoveContainer" containerID="f6f8553d16b6bf4411be31a38d3f7f4e1b7cb37e49e6134030e109e8a43ddc80" Oct 30 20:15:03.989581 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:15:03.989557 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-389c4e-29889855-lbsk6_odh-ai-gateway-infra(219a0a4f-88b9-4f4c-92c6-4a8ffa788842)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-389c4e-29889855-lbsk6" podUID="219a0a4f-88b9-4f4c-92c6-4a8ffa788842" Oct 30 20:15:03.991514 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:03.991261 4292 generic.go:356] "Generic (PLEG): container finished" podID="6c74471c-9ea5-486f-b083-efee6a92777d" containerID="f132942b5f544821944ddce0d13c5a1b9ca92c5389a84a6193cb187dc9cf8af9" exitCode=60 Oct 30 20:15:03.991514 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:03.991304 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889855-j5krt" event={"ID":"6c74471c-9ea5-486f-b083-efee6a92777d","Type":"ContainerDied","Data":"f132942b5f544821944ddce0d13c5a1b9ca92c5389a84a6193cb187dc9cf8af9"} Oct 30 20:15:03.991641 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:03.991558 4292 scope.go:122] "RemoveContainer" containerID="f132942b5f544821944ddce0d13c5a1b9ca92c5389a84a6193cb187dc9cf8af9" Oct 30 20:15:03.991801 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:15:03.991782 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-4ec609-29889855-j5krt_odh-ai-gateway-infra(6c74471c-9ea5-486f-b083-efee6a92777d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889855-j5krt" podUID="6c74471c-9ea5-486f-b083-efee6a92777d" Oct 30 20:15:03.993454 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:03.993432 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht" event={"ID":"8f9dd92b-3ba4-4d87-8162-df97a30a48d0","Type":"ContainerStarted","Data":"4f7a5376f0f046577e9750c1028030d6002a9b911bacc4cdc3a383fd027af037"} Oct 30 20:15:03.993608 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:03.993590 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht" event={"ID":"8f9dd92b-3ba4-4d87-8162-df97a30a48d0","Type":"ContainerStarted","Data":"513e7ad811b4c198baa46c0390dd393155127179d2707603b70e1ed15f4fcdef"} Oct 30 20:15:03.995608 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:03.995588 4292 scope.go:122] "RemoveContainer" containerID="8b7acbe7c84d4db2358e3cfd9df5ac0e0d49a40fcf822f86ffad7acde9331dcc" Oct 30 20:15:03.995799 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:15:03.995779 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-d5d3ba-29889855-bd72m_odh-ai-gateway-infra(0a4c41bd-fc28-4552-be12-170b75585dae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d5d3ba-29889855-bd72m" podUID="0a4c41bd-fc28-4552-be12-170b75585dae" Oct 30 20:15:03.997257 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:03.997235 4292 generic.go:356] "Generic (PLEG): container finished" podID="c4374031-874c-47dd-b5a0-4bbd49dde2ba" containerID="b97450241ed4ebcd7b191d65370b987fd11ff0b5826acb56bec88465fb625d1d" exitCode=0 Oct 30 20:15:03.997408 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:03.997299 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c6d29b-htqdr" event={"ID":"c4374031-874c-47dd-b5a0-4bbd49dde2ba","Type":"ContainerDied","Data":"b97450241ed4ebcd7b191d65370b987fd11ff0b5826acb56bec88465fb625d1d"} Oct 30 20:15:04.011450 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:04.011429 4292 scope.go:122] "RemoveContainer" containerID="37d47786eba1c955d791c2b2a4d1c7fd12f14da9d3add3f7e9f78af3c83840d2" Oct 30 20:15:04.580707 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:04.580674 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht" Oct 30 20:15:04.587291 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:04.587261 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 30 20:15:04.587518 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:04.587490 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht" podUID="8f9dd92b-3ba4-4d87-8162-df97a30a48d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 30 20:15:04.699869 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:04.699839 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-389c4e-29889855-lbsk6"] Oct 30 20:15:04.719866 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:04.719804 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht" podStartSLOduration=2.719786979 podStartE2EDuration="2.719786979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:04.715224709 +0000 UTC m=+3008.233540192" watchObservedRunningTime="2026-10-30 20:15:04.719786979 +0000 UTC m=+3008.238102438" Oct 30 20:15:05.011883 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:05.011805 4292 scope.go:122] "RemoveContainer" containerID="f6f8553d16b6bf4411be31a38d3f7f4e1b7cb37e49e6134030e109e8a43ddc80" Oct 30 20:15:05.013195 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:15:05.013159 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-389c4e-29889855-lbsk6_odh-ai-gateway-infra(219a0a4f-88b9-4f4c-92c6-4a8ffa788842)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-389c4e-29889855-lbsk6" podUID="219a0a4f-88b9-4f4c-92c6-4a8ffa788842" Oct 30 20:15:05.016666 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:05.016639 4292 scope.go:122] "RemoveContainer" containerID="f132942b5f544821944ddce0d13c5a1b9ca92c5389a84a6193cb187dc9cf8af9" Oct 30 20:15:05.017668 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:15:05.017643 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-4ec609-29889855-j5krt_odh-ai-gateway-infra(6c74471c-9ea5-486f-b083-efee6a92777d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889855-j5krt" podUID="6c74471c-9ea5-486f-b083-efee6a92777d" Oct 30 20:15:05.203276 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:05.203252 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c6d29b-htqdr" Oct 30 20:15:05.311077 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:05.311048 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4374031-874c-47dd-b5a0-4bbd49dde2ba-kube-api-access-mlxfb\" (UniqueName: \"kubernetes.io/projected/c4374031-874c-47dd-b5a0-4bbd49dde2ba-kube-api-access-mlxfb\") pod \"c4374031-874c-47dd-b5a0-4bbd49dde2ba\" (UID: \"c4374031-874c-47dd-b5a0-4bbd49dde2ba\") " Oct 30 20:15:05.311255 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:05.311093 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4374031-874c-47dd-b5a0-4bbd49dde2ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4374031-874c-47dd-b5a0-4bbd49dde2ba-maas-api-openshift-ca\") pod \"c4374031-874c-47dd-b5a0-4bbd49dde2ba\" (UID: \"c4374031-874c-47dd-b5a0-4bbd49dde2ba\") " Oct 30 20:15:05.311255 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:05.311142 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4374031-874c-47dd-b5a0-4bbd49dde2ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4374031-874c-47dd-b5a0-4bbd49dde2ba-maas-api-xks-ca\") pod \"c4374031-874c-47dd-b5a0-4bbd49dde2ba\" (UID: \"c4374031-874c-47dd-b5a0-4bbd49dde2ba\") " Oct 30 20:15:05.311571 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:05.311545 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4374031-874c-47dd-b5a0-4bbd49dde2ba-maas-api-openshift-ca" pod "c4374031-874c-47dd-b5a0-4bbd49dde2ba" (UID: "c4374031-874c-47dd-b5a0-4bbd49dde2ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:05.311661 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:05.311648 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4374031-874c-47dd-b5a0-4bbd49dde2ba-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:05.313194 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:05.313169 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4374031-874c-47dd-b5a0-4bbd49dde2ba-maas-api-xks-ca" pod "c4374031-874c-47dd-b5a0-4bbd49dde2ba" (UID: "c4374031-874c-47dd-b5a0-4bbd49dde2ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:15:05.313297 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:05.313198 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4374031-874c-47dd-b5a0-4bbd49dde2ba-kube-api-access-mlxfb" pod "c4374031-874c-47dd-b5a0-4bbd49dde2ba" (UID: "c4374031-874c-47dd-b5a0-4bbd49dde2ba"). InnerVolumeSpecName "kube-api-access-mlxfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:05.412901 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:05.412868 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlxfb\" (UniqueName: \"kubernetes.io/projected/c4374031-874c-47dd-b5a0-4bbd49dde2ba-kube-api-access-mlxfb\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:05.412901 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:05.412898 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4374031-874c-47dd-b5a0-4bbd49dde2ba-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:05.577076 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:05.576635 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 30 20:15:05.577076 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:05.576685 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht" podUID="8f9dd92b-3ba4-4d87-8162-df97a30a48d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 30 20:15:06.024046 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:06.024021 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c6d29b-htqdr" Oct 30 20:15:06.024569 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:06.024011 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c6d29b-htqdr" event={"ID":"c4374031-874c-47dd-b5a0-4bbd49dde2ba","Type":"ContainerDied","Data":"67cf008bb7683899e421717becd40bbc11f42284013cb64cec6c54488e189ba7"} Oct 30 20:15:06.024569 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:06.024149 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67cf008bb7683899e421717becd40bbc11f42284013cb64cec6c54488e189ba7" Oct 30 20:15:06.285613 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:06.285546 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-389c4e-29889855-lbsk6" Oct 30 20:15:06.424565 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:06.424526 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/219a0a4f-88b9-4f4c-92c6-4a8ffa788842-kube-api-access-h445p\" (UniqueName: \"kubernetes.io/projected/219a0a4f-88b9-4f4c-92c6-4a8ffa788842-kube-api-access-h445p\") pod \"219a0a4f-88b9-4f4c-92c6-4a8ffa788842\" (UID: \"219a0a4f-88b9-4f4c-92c6-4a8ffa788842\") " Oct 30 20:15:06.424724 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:06.424625 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/219a0a4f-88b9-4f4c-92c6-4a8ffa788842-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/219a0a4f-88b9-4f4c-92c6-4a8ffa788842-maas-api-service-ca\") pod \"219a0a4f-88b9-4f4c-92c6-4a8ffa788842\" (UID: \"219a0a4f-88b9-4f4c-92c6-4a8ffa788842\") " Oct 30 20:15:06.425011 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:06.424984 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/219a0a4f-88b9-4f4c-92c6-4a8ffa788842-maas-api-service-ca" pod "219a0a4f-88b9-4f4c-92c6-4a8ffa788842" (UID: "219a0a4f-88b9-4f4c-92c6-4a8ffa788842"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:06.425213 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:06.425196 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/219a0a4f-88b9-4f4c-92c6-4a8ffa788842-maas-api-service-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:06.426671 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:06.426649 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/219a0a4f-88b9-4f4c-92c6-4a8ffa788842-kube-api-access-h445p" pod "219a0a4f-88b9-4f4c-92c6-4a8ffa788842" (UID: "219a0a4f-88b9-4f4c-92c6-4a8ffa788842"). InnerVolumeSpecName "kube-api-access-h445p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:06.526620 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:06.526589 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h445p\" (UniqueName: \"kubernetes.io/projected/219a0a4f-88b9-4f4c-92c6-4a8ffa788842-kube-api-access-h445p\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:06.594428 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:06.586763 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 30 20:15:06.594428 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:06.586823 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht" podUID="8f9dd92b-3ba4-4d87-8162-df97a30a48d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 30 20:15:06.776112 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:06.776073 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-389c4e-966lm"] Oct 30 20:15:06.776282 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:06.776138 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-389c4e-966lm"] Oct 30 20:15:07.032030 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:07.031999 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-389c4e-29889855-lbsk6" Oct 30 20:15:07.032470 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:07.032030 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-389c4e-29889855-lbsk6" event={"ID":"219a0a4f-88b9-4f4c-92c6-4a8ffa788842","Type":"ContainerDied","Data":"96c00913f076c959f21beddb714c5b6d12773415b0b41bf24b948d6036602840"} Oct 30 20:15:07.032470 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:07.032096 4292 scope.go:122] "RemoveContainer" containerID="f6f8553d16b6bf4411be31a38d3f7f4e1b7cb37e49e6134030e109e8a43ddc80" Oct 30 20:15:07.200962 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:07.200924 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-389c4e-29889855-lbsk6"] Oct 30 20:15:07.205383 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:07.205346 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-389c4e-29889855-lbsk6"] Oct 30 20:15:07.577112 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:07.577074 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 30 20:15:07.577309 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:07.577146 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht" podUID="8f9dd92b-3ba4-4d87-8162-df97a30a48d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 30 20:15:08.576766 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:08.576726 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 30 20:15:08.577270 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:08.576791 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht" podUID="8f9dd92b-3ba4-4d87-8162-df97a30a48d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 30 20:15:08.585425 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:08.585393 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="219a0a4f-88b9-4f4c-92c6-4a8ffa788842" path="/var/lib/kubelet/pods/219a0a4f-88b9-4f4c-92c6-4a8ffa788842/volumes" Oct 30 20:15:08.585863 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:08.585849 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db133a39-3ac3-49de-91e9-6eac2647b59a" path="/var/lib/kubelet/pods/db133a39-3ac3-49de-91e9-6eac2647b59a/volumes" Oct 30 20:15:09.576407 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:09.576373 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 30 20:15:09.576612 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:09.576428 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht" podUID="8f9dd92b-3ba4-4d87-8162-df97a30a48d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 30 20:15:10.582952 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:10.582872 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht" Oct 30 20:15:10.583443 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:10.583264 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht" Oct 30 20:15:10.583942 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:10.583927 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht" Oct 30 20:15:12.063523 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:12.063491 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-389c4e/praxis-route-389c4e-kserve-7788b879cc-585s7"] Oct 30 20:15:13.140460 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:13.140428 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr"] Oct 30 20:15:13.140887 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:13.140875 4292 cpu_manager.go:414] "containerMap: removing container" podUID="c4374031-874c-47dd-b5a0-4bbd49dde2ba" containerName="revoke-keys" Oct 30 20:15:13.140932 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:13.140888 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4374031-874c-47dd-b5a0-4bbd49dde2ba" containerName="revoke-keys" Oct 30 20:15:13.140932 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:13.140921 4292 cpu_manager.go:414] "containerMap: removing container" podUID="219a0a4f-88b9-4f4c-92c6-4a8ffa788842" containerName="cleanup" Oct 30 20:15:13.140932 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:13.140926 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="219a0a4f-88b9-4f4c-92c6-4a8ffa788842" containerName="cleanup" Oct 30 20:15:13.141027 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:13.140935 4292 cpu_manager.go:414] "containerMap: removing container" podUID="219a0a4f-88b9-4f4c-92c6-4a8ffa788842" containerName="cleanup" Oct 30 20:15:13.141027 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:13.140939 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="219a0a4f-88b9-4f4c-92c6-4a8ffa788842" containerName="cleanup" Oct 30 20:15:13.141027 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:13.141004 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="219a0a4f-88b9-4f4c-92c6-4a8ffa788842" containerName="cleanup" Oct 30 20:15:13.141027 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:13.141012 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="219a0a4f-88b9-4f4c-92c6-4a8ffa788842" containerName="cleanup" Oct 30 20:15:13.141027 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:13.141017 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4374031-874c-47dd-b5a0-4bbd49dde2ba" containerName="revoke-keys" Oct 30 20:15:13.145860 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:13.145839 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr" Oct 30 20:15:13.212323 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:13.212287 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr"] Oct 30 20:15:13.228568 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:15:13.228526 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:15:13.251013 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:13.250982 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-a74d15-openshift-default-dockercfg-pfbt9\"" Oct 30 20:15:13.320856 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:13.320819 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/623dc6cb-9a97-48df-9388-cf847d18a146-istio-data\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr\" (UID: \"623dc6cb-9a97-48df-9388-cf847d18a146\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr" Oct 30 20:15:13.320856 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:13.320859 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/623dc6cb-9a97-48df-9388-cf847d18a146-istio-envoy\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr\" (UID: \"623dc6cb-9a97-48df-9388-cf847d18a146\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr" Oct 30 20:15:13.321079 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:13.320992 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/623dc6cb-9a97-48df-9388-cf847d18a146-istiod-ca-cert\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr\" (UID: \"623dc6cb-9a97-48df-9388-cf847d18a146\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr" Oct 30 20:15:13.321079 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:13.321013 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/623dc6cb-9a97-48df-9388-cf847d18a146-istio-podinfo\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr\" (UID: \"623dc6cb-9a97-48df-9388-cf847d18a146\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr" Oct 30 20:15:13.321079 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:13.321047 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/623dc6cb-9a97-48df-9388-cf847d18a146-credential-socket\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr\" (UID: \"623dc6cb-9a97-48df-9388-cf847d18a146\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr" Oct 30 20:15:13.321079 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:13.321074 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/623dc6cb-9a97-48df-9388-cf847d18a146-istio-token\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr\" (UID: \"623dc6cb-9a97-48df-9388-cf847d18a146\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr" Oct 30 20:15:13.321234 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:13.321138 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2dvc\" (UniqueName: \"kubernetes.io/projected/623dc6cb-9a97-48df-9388-cf847d18a146-kube-api-access-g2dvc\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr\" (UID: \"623dc6cb-9a97-48df-9388-cf847d18a146\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr" Oct 30 20:15:13.321234 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:13.321176 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/623dc6cb-9a97-48df-9388-cf847d18a146-workload-socket\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr\" (UID: \"623dc6cb-9a97-48df-9388-cf847d18a146\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr" Oct 30 20:15:13.321234 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:13.321191 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/623dc6cb-9a97-48df-9388-cf847d18a146-workload-certs\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr\" (UID: \"623dc6cb-9a97-48df-9388-cf847d18a146\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr" Oct 30 20:15:13.422656 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:13.422534 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/623dc6cb-9a97-48df-9388-cf847d18a146-istiod-ca-cert\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr\" (UID: \"623dc6cb-9a97-48df-9388-cf847d18a146\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr" Oct 30 20:15:13.422656 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:13.422586 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/623dc6cb-9a97-48df-9388-cf847d18a146-istio-podinfo\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr\" (UID: \"623dc6cb-9a97-48df-9388-cf847d18a146\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr" Oct 30 20:15:13.422656 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:13.422639 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/623dc6cb-9a97-48df-9388-cf847d18a146-credential-socket\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr\" (UID: \"623dc6cb-9a97-48df-9388-cf847d18a146\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr" Oct 30 20:15:13.422938 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:13.422829 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/623dc6cb-9a97-48df-9388-cf847d18a146-istio-token\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr\" (UID: \"623dc6cb-9a97-48df-9388-cf847d18a146\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr" Oct 30 20:15:13.422938 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:13.422914 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2dvc\" (UniqueName: \"kubernetes.io/projected/623dc6cb-9a97-48df-9388-cf847d18a146-kube-api-access-g2dvc\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr\" (UID: \"623dc6cb-9a97-48df-9388-cf847d18a146\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr" Oct 30 20:15:13.423036 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:13.422961 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/623dc6cb-9a97-48df-9388-cf847d18a146-workload-socket\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr\" (UID: \"623dc6cb-9a97-48df-9388-cf847d18a146\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr" Oct 30 20:15:13.423036 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:13.422973 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/623dc6cb-9a97-48df-9388-cf847d18a146-credential-socket\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr\" (UID: \"623dc6cb-9a97-48df-9388-cf847d18a146\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr" Oct 30 20:15:13.423036 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:13.422983 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/623dc6cb-9a97-48df-9388-cf847d18a146-workload-certs\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr\" (UID: \"623dc6cb-9a97-48df-9388-cf847d18a146\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr" Oct 30 20:15:13.423202 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:13.423087 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/623dc6cb-9a97-48df-9388-cf847d18a146-istio-data\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr\" (UID: \"623dc6cb-9a97-48df-9388-cf847d18a146\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr" Oct 30 20:15:13.423202 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:13.423116 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/623dc6cb-9a97-48df-9388-cf847d18a146-istio-envoy\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr\" (UID: \"623dc6cb-9a97-48df-9388-cf847d18a146\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr" Oct 30 20:15:13.423292 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:13.423262 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/623dc6cb-9a97-48df-9388-cf847d18a146-workload-certs\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr\" (UID: \"623dc6cb-9a97-48df-9388-cf847d18a146\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr" Oct 30 20:15:13.423365 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:13.423310 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/623dc6cb-9a97-48df-9388-cf847d18a146-workload-socket\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr\" (UID: \"623dc6cb-9a97-48df-9388-cf847d18a146\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr" Oct 30 20:15:13.423471 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:13.423448 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/623dc6cb-9a97-48df-9388-cf847d18a146-istio-data\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr\" (UID: \"623dc6cb-9a97-48df-9388-cf847d18a146\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr" Oct 30 20:15:13.423525 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:13.423478 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/623dc6cb-9a97-48df-9388-cf847d18a146-istiod-ca-cert\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr\" (UID: \"623dc6cb-9a97-48df-9388-cf847d18a146\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr" Oct 30 20:15:13.425216 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:13.425190 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/623dc6cb-9a97-48df-9388-cf847d18a146-istio-podinfo\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr\" (UID: \"623dc6cb-9a97-48df-9388-cf847d18a146\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr" Oct 30 20:15:13.425345 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:13.425196 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/623dc6cb-9a97-48df-9388-cf847d18a146-istio-envoy\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr\" (UID: \"623dc6cb-9a97-48df-9388-cf847d18a146\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr" Oct 30 20:15:13.489936 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:13.489908 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/623dc6cb-9a97-48df-9388-cf847d18a146-istio-token\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr\" (UID: \"623dc6cb-9a97-48df-9388-cf847d18a146\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr" Oct 30 20:15:13.490684 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:13.490656 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2dvc\" (UniqueName: \"kubernetes.io/projected/623dc6cb-9a97-48df-9388-cf847d18a146-kube-api-access-g2dvc\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr\" (UID: \"623dc6cb-9a97-48df-9388-cf847d18a146\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr" Oct 30 20:15:13.576547 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:13.576516 4292 scope.go:122] "RemoveContainer" containerID="f132942b5f544821944ddce0d13c5a1b9ca92c5389a84a6193cb187dc9cf8af9" Oct 30 20:15:13.759652 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:13.759572 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr" Oct 30 20:15:14.080814 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:14.080776 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889855-j5krt" event={"ID":"6c74471c-9ea5-486f-b083-efee6a92777d","Type":"ContainerStarted","Data":"28ff788553b86ce147c00d65c7c51ce482865c196dd70b3b58888e0a6889940b"} Oct 30 20:15:14.186293 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:14.186259 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr"] Oct 30 20:15:14.215069 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:15:14.215040 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod623dc6cb_9a97_48df_9388_cf847d18a146.slice/crio-a79dc821a5b3dde6f08d5b6a73236af8f4afa784601fa801aa92298ff63ecf08 WatchSource:0}: Error finding container a79dc821a5b3dde6f08d5b6a73236af8f4afa784601fa801aa92298ff63ecf08: Status 404 returned error can't find the container with id a79dc821a5b3dde6f08d5b6a73236af8f4afa784601fa801aa92298ff63ecf08 Oct 30 20:15:14.217844 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:14.217805 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:15:14.217926 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:14.217900 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:15:14.217977 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:14.217946 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:15:15.095263 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:15.095226 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr" event={"ID":"623dc6cb-9a97-48df-9388-cf847d18a146","Type":"ContainerStarted","Data":"6d199db1668d4b9305b9fa05db8bc858682972d13cf53698d367c4f4ac7fffd3"} Oct 30 20:15:15.095480 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:15.095292 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr" event={"ID":"623dc6cb-9a97-48df-9388-cf847d18a146","Type":"ContainerStarted","Data":"a79dc821a5b3dde6f08d5b6a73236af8f4afa784601fa801aa92298ff63ecf08"} Oct 30 20:15:15.097246 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:15.097218 4292 generic.go:356] "Generic (PLEG): container finished" podID="6c74471c-9ea5-486f-b083-efee6a92777d" containerID="28ff788553b86ce147c00d65c7c51ce482865c196dd70b3b58888e0a6889940b" exitCode=60 Oct 30 20:15:15.097414 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:15.097252 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889855-j5krt" event={"ID":"6c74471c-9ea5-486f-b083-efee6a92777d","Type":"ContainerDied","Data":"28ff788553b86ce147c00d65c7c51ce482865c196dd70b3b58888e0a6889940b"} Oct 30 20:15:15.097414 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:15.097300 4292 scope.go:122] "RemoveContainer" containerID="f132942b5f544821944ddce0d13c5a1b9ca92c5389a84a6193cb187dc9cf8af9" Oct 30 20:15:15.097701 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:15.097678 4292 scope.go:122] "RemoveContainer" containerID="28ff788553b86ce147c00d65c7c51ce482865c196dd70b3b58888e0a6889940b" Oct 30 20:15:15.098011 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:15:15.097986 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-4ec609-29889855-j5krt_odh-ai-gateway-infra(6c74471c-9ea5-486f-b083-efee6a92777d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889855-j5krt" podUID="6c74471c-9ea5-486f-b083-efee6a92777d" Oct 30 20:15:15.210283 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:15.210249 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889855-j5krt"] Oct 30 20:15:15.275312 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:15.275238 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr" podStartSLOduration=2.275197856 podStartE2EDuration="2.275197856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:15.265137505 +0000 UTC m=+3018.783452961" watchObservedRunningTime="2026-10-30 20:15:15.275197856 +0000 UTC m=+3018.793513312" Oct 30 20:15:15.597624 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:15.597579 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5"] Oct 30 20:15:15.597885 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:15.597855 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5" podUID="c6d51da1-a6c7-4060-8da8-74b8b56e949d" containerName="istio-proxy" containerID="cri-o://84b12bffa6d75a2e202ac3503c8a6b58f309b6e7334c3e86114db9acde76dc6d" gracePeriod=30 Oct 30 20:15:15.762786 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:15.762710 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr" Oct 30 20:15:15.774499 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:15.774462 4292 patch_prober.go:29] interesting pod/e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Oct 30 20:15:15.774765 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:15.774742 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr" podUID="623dc6cb-9a97-48df-9388-cf847d18a146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Oct 30 20:15:16.281165 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:16.281136 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889855-j5krt" Oct 30 20:15:16.376910 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:16.376812 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c74471c-9ea5-486f-b083-efee6a92777d-kube-api-access-zfdkz\" (UniqueName: \"kubernetes.io/projected/6c74471c-9ea5-486f-b083-efee6a92777d-kube-api-access-zfdkz\") pod \"6c74471c-9ea5-486f-b083-efee6a92777d\" (UID: \"6c74471c-9ea5-486f-b083-efee6a92777d\") " Oct 30 20:15:16.377082 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:16.376945 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c74471c-9ea5-486f-b083-efee6a92777d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c74471c-9ea5-486f-b083-efee6a92777d-maas-api-service-ca\") pod \"6c74471c-9ea5-486f-b083-efee6a92777d\" (UID: \"6c74471c-9ea5-486f-b083-efee6a92777d\") " Oct 30 20:15:16.377482 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:16.377448 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c74471c-9ea5-486f-b083-efee6a92777d-maas-api-service-ca" pod "6c74471c-9ea5-486f-b083-efee6a92777d" (UID: "6c74471c-9ea5-486f-b083-efee6a92777d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:16.379568 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:16.379542 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c74471c-9ea5-486f-b083-efee6a92777d-kube-api-access-zfdkz" pod "6c74471c-9ea5-486f-b083-efee6a92777d" (UID: "6c74471c-9ea5-486f-b083-efee6a92777d"). InnerVolumeSpecName "kube-api-access-zfdkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:16.481309 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:16.481258 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfdkz\" (UniqueName: \"kubernetes.io/projected/6c74471c-9ea5-486f-b083-efee6a92777d-kube-api-access-zfdkz\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:16.481309 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:16.481308 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c74471c-9ea5-486f-b083-efee6a92777d-maas-api-service-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:16.583418 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:16.583387 4292 scope.go:122] "RemoveContainer" containerID="8b7acbe7c84d4db2358e3cfd9df5ac0e0d49a40fcf822f86ffad7acde9331dcc" Oct 30 20:15:16.583664 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:16.583641 4292 scope.go:122] "RemoveContainer" containerID="e99ae5bcd1c114b26a677d4846bb4eb3743108b19046574c05903820ebccd0e2" Oct 30 20:15:16.770377 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:16.765753 4292 patch_prober.go:29] interesting pod/e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Oct 30 20:15:16.770377 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:16.765811 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr" podUID="623dc6cb-9a97-48df-9388-cf847d18a146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Oct 30 20:15:17.130290 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:17.130256 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889855-s4n4l" event={"ID":"e455047b-0948-4538-a504-9e0a3324b00c","Type":"ContainerStarted","Data":"447d4637b99d888ea916f0d2e0d2e17c840e61ee75a6a10a512e9a5be9380a87"} Oct 30 20:15:17.131737 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:17.131713 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889855-j5krt" Oct 30 20:15:17.131883 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:17.131755 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889855-j5krt" event={"ID":"6c74471c-9ea5-486f-b083-efee6a92777d","Type":"ContainerDied","Data":"72835a2ddaeb42d5561c1a9f7fbab3ba002e845666f3b4c3d23a2952e0ddf736"} Oct 30 20:15:17.131883 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:17.131808 4292 scope.go:122] "RemoveContainer" containerID="28ff788553b86ce147c00d65c7c51ce482865c196dd70b3b58888e0a6889940b" Oct 30 20:15:17.134115 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:17.134090 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d5d3ba-29889855-bd72m" event={"ID":"0a4c41bd-fc28-4552-be12-170b75585dae","Type":"ContainerStarted","Data":"b49acb33a8a393163a317e411fec9e95cfefbba9a229ba90a2450f0e12493bd5"} Oct 30 20:15:17.249322 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:17.249222 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889855-s4n4l" podStartSLOduration=17.249207082 podStartE2EDuration="17.249207082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:17.24829584 +0000 UTC m=+3020.766611309" watchObservedRunningTime="2026-10-30 20:15:17.249207082 +0000 UTC m=+3020.767522537" Oct 30 20:15:17.379999 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:17.379961 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889855-j5krt"] Oct 30 20:15:17.382972 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:17.382942 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ec609-29889855-j5krt"] Oct 30 20:15:17.487893 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:17.487838 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d5d3ba-29889855-bd72m" podStartSLOduration=17.487823487 podStartE2EDuration="17.487823487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:17.487272846 +0000 UTC m=+3021.005588320" watchObservedRunningTime="2026-10-30 20:15:17.487823487 +0000 UTC m=+3021.006138985" Oct 30 20:15:17.772376 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:17.771741 4292 patch_prober.go:29] interesting pod/e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Oct 30 20:15:17.772376 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:17.771803 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr" podUID="623dc6cb-9a97-48df-9388-cf847d18a146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Oct 30 20:15:18.142631 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:18.142594 4292 generic.go:356] "Generic (PLEG): container finished" podID="e455047b-0948-4538-a504-9e0a3324b00c" containerID="447d4637b99d888ea916f0d2e0d2e17c840e61ee75a6a10a512e9a5be9380a87" exitCode=60 Oct 30 20:15:18.142831 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:18.142665 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889855-s4n4l" event={"ID":"e455047b-0948-4538-a504-9e0a3324b00c","Type":"ContainerDied","Data":"447d4637b99d888ea916f0d2e0d2e17c840e61ee75a6a10a512e9a5be9380a87"} Oct 30 20:15:18.142831 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:18.142735 4292 scope.go:122] "RemoveContainer" containerID="e99ae5bcd1c114b26a677d4846bb4eb3743108b19046574c05903820ebccd0e2" Oct 30 20:15:18.143135 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:18.143116 4292 scope.go:122] "RemoveContainer" containerID="447d4637b99d888ea916f0d2e0d2e17c840e61ee75a6a10a512e9a5be9380a87" Oct 30 20:15:18.143468 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:15:18.143444 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889855-s4n4l_odh-ai-gateway-infra(e455047b-0948-4538-a504-9e0a3324b00c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889855-s4n4l" podUID="e455047b-0948-4538-a504-9e0a3324b00c" Oct 30 20:15:18.145832 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:18.145812 4292 generic.go:356] "Generic (PLEG): container finished" podID="0a4c41bd-fc28-4552-be12-170b75585dae" containerID="b49acb33a8a393163a317e411fec9e95cfefbba9a229ba90a2450f0e12493bd5" exitCode=60 Oct 30 20:15:18.145913 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:18.145870 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d5d3ba-29889855-bd72m" event={"ID":"0a4c41bd-fc28-4552-be12-170b75585dae","Type":"ContainerDied","Data":"b49acb33a8a393163a317e411fec9e95cfefbba9a229ba90a2450f0e12493bd5"} Oct 30 20:15:18.146191 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:18.146178 4292 scope.go:122] "RemoveContainer" containerID="b49acb33a8a393163a317e411fec9e95cfefbba9a229ba90a2450f0e12493bd5" Oct 30 20:15:18.146444 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:15:18.146426 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-d5d3ba-29889855-bd72m_odh-ai-gateway-infra(0a4c41bd-fc28-4552-be12-170b75585dae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d5d3ba-29889855-bd72m" podUID="0a4c41bd-fc28-4552-be12-170b75585dae" Oct 30 20:15:18.177813 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:18.177790 4292 scope.go:122] "RemoveContainer" containerID="8b7acbe7c84d4db2358e3cfd9df5ac0e0d49a40fcf822f86ffad7acde9331dcc" Oct 30 20:15:18.275531 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:18.275488 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889855-s4n4l"] Oct 30 20:15:18.498886 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:18.498807 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d5d3ba-29889855-bd72m"] Oct 30 20:15:18.583222 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:18.583181 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c74471c-9ea5-486f-b083-efee6a92777d" path="/var/lib/kubelet/pods/6c74471c-9ea5-486f-b083-efee6a92777d/volumes" Oct 30 20:15:18.609885 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:18.609848 4292 patch_prober.go:29] interesting pod/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:15:18.610026 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:18.609907 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5" podUID="c6d51da1-a6c7-4060-8da8-74b8b56e949d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:15:18.761930 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:18.761837 4292 patch_prober.go:29] interesting pod/e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Oct 30 20:15:18.762224 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:18.762185 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr" podUID="623dc6cb-9a97-48df-9388-cf847d18a146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Oct 30 20:15:19.339694 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:19.339663 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889855-s4n4l" Oct 30 20:15:19.388365 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:19.388328 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d5d3ba-29889855-bd72m" Oct 30 20:15:19.515151 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:19.515033 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e455047b-0948-4538-a504-9e0a3324b00c-kube-api-access-d2zqf\" (UniqueName: \"kubernetes.io/projected/e455047b-0948-4538-a504-9e0a3324b00c-kube-api-access-d2zqf\") pod \"e455047b-0948-4538-a504-9e0a3324b00c\" (UID: \"e455047b-0948-4538-a504-9e0a3324b00c\") " Oct 30 20:15:19.515151 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:19.515127 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e455047b-0948-4538-a504-9e0a3324b00c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e455047b-0948-4538-a504-9e0a3324b00c-maas-api-service-ca\") pod \"e455047b-0948-4538-a504-9e0a3324b00c\" (UID: \"e455047b-0948-4538-a504-9e0a3324b00c\") " Oct 30 20:15:19.515659 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:19.515256 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a4c41bd-fc28-4552-be12-170b75585dae-kube-api-access-jgsln\" (UniqueName: \"kubernetes.io/projected/0a4c41bd-fc28-4552-be12-170b75585dae-kube-api-access-jgsln\") pod \"0a4c41bd-fc28-4552-be12-170b75585dae\" (UID: \"0a4c41bd-fc28-4552-be12-170b75585dae\") " Oct 30 20:15:19.515659 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:19.515313 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a4c41bd-fc28-4552-be12-170b75585dae-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a4c41bd-fc28-4552-be12-170b75585dae-maas-api-service-ca\") pod \"0a4c41bd-fc28-4552-be12-170b75585dae\" (UID: \"0a4c41bd-fc28-4552-be12-170b75585dae\") " Oct 30 20:15:19.515659 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:19.515565 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e455047b-0948-4538-a504-9e0a3324b00c-maas-api-service-ca" pod "e455047b-0948-4538-a504-9e0a3324b00c" (UID: "e455047b-0948-4538-a504-9e0a3324b00c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:19.515659 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:19.515635 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a4c41bd-fc28-4552-be12-170b75585dae-maas-api-service-ca" pod "0a4c41bd-fc28-4552-be12-170b75585dae" (UID: "0a4c41bd-fc28-4552-be12-170b75585dae"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:19.515793 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:19.515773 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e455047b-0948-4538-a504-9e0a3324b00c-maas-api-service-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:19.515793 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:19.515790 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a4c41bd-fc28-4552-be12-170b75585dae-maas-api-service-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:19.517201 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:19.517177 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e455047b-0948-4538-a504-9e0a3324b00c-kube-api-access-d2zqf" pod "e455047b-0948-4538-a504-9e0a3324b00c" (UID: "e455047b-0948-4538-a504-9e0a3324b00c"). InnerVolumeSpecName "kube-api-access-d2zqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:19.517201 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:19.517189 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a4c41bd-fc28-4552-be12-170b75585dae-kube-api-access-jgsln" pod "0a4c41bd-fc28-4552-be12-170b75585dae" (UID: "0a4c41bd-fc28-4552-be12-170b75585dae"). InnerVolumeSpecName "kube-api-access-jgsln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:19.617033 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:19.616994 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2zqf\" (UniqueName: \"kubernetes.io/projected/e455047b-0948-4538-a504-9e0a3324b00c-kube-api-access-d2zqf\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:19.617033 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:19.617026 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgsln\" (UniqueName: \"kubernetes.io/projected/0a4c41bd-fc28-4552-be12-170b75585dae-kube-api-access-jgsln\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:19.774776 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:19.774741 4292 patch_prober.go:29] interesting pod/e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Oct 30 20:15:19.774994 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:19.774799 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr" podUID="623dc6cb-9a97-48df-9388-cf847d18a146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Oct 30 20:15:20.186240 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:20.186138 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d5d3ba-29889855-bd72m" Oct 30 20:15:20.186240 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:20.186172 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d5d3ba-29889855-bd72m" event={"ID":"0a4c41bd-fc28-4552-be12-170b75585dae","Type":"ContainerDied","Data":"03254b53705263eb8d4e89de2e51b21e31b8b539c749186681c01f150f37c9db"} Oct 30 20:15:20.186490 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:20.186255 4292 scope.go:122] "RemoveContainer" containerID="b49acb33a8a393163a317e411fec9e95cfefbba9a229ba90a2450f0e12493bd5" Oct 30 20:15:20.188257 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:20.187971 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889855-s4n4l" event={"ID":"e455047b-0948-4538-a504-9e0a3324b00c","Type":"ContainerDied","Data":"02f76cf77a781c0918307a9e284ea20ddd7701c1dcd42ccca29077f6ba429819"} Oct 30 20:15:20.188257 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:20.188045 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889855-s4n4l" Oct 30 20:15:20.204620 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:20.204600 4292 scope.go:122] "RemoveContainer" containerID="447d4637b99d888ea916f0d2e0d2e17c840e61ee75a6a10a512e9a5be9380a87" Oct 30 20:15:20.358677 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:20.358650 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d5d3ba-29889855-bd72m"] Oct 30 20:15:20.361372 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:20.361339 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d5d3ba-29889855-bd72m"] Oct 30 20:15:20.476039 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:20.475958 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889855-s4n4l"] Oct 30 20:15:20.481911 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:20.481886 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889855-s4n4l"] Oct 30 20:15:20.586151 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:20.586113 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a4c41bd-fc28-4552-be12-170b75585dae" path="/var/lib/kubelet/pods/0a4c41bd-fc28-4552-be12-170b75585dae/volumes" Oct 30 20:15:20.586685 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:20.586595 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e455047b-0948-4538-a504-9e0a3324b00c" path="/var/lib/kubelet/pods/e455047b-0948-4538-a504-9e0a3324b00c/volumes" Oct 30 20:15:20.760858 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:20.760773 4292 patch_prober.go:29] interesting pod/e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Oct 30 20:15:20.760858 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:20.760828 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr" podUID="623dc6cb-9a97-48df-9388-cf847d18a146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Oct 30 20:15:20.849676 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:20.849651 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5" Oct 30 20:15:21.031161 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:21.031125 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6d51da1-a6c7-4060-8da8-74b8b56e949d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6d51da1-a6c7-4060-8da8-74b8b56e949d-credential-socket\") pod \"c6d51da1-a6c7-4060-8da8-74b8b56e949d\" (UID: \"c6d51da1-a6c7-4060-8da8-74b8b56e949d\") " Oct 30 20:15:21.031372 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:21.031192 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6d51da1-a6c7-4060-8da8-74b8b56e949d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6d51da1-a6c7-4060-8da8-74b8b56e949d-istio-data\") pod \"c6d51da1-a6c7-4060-8da8-74b8b56e949d\" (UID: \"c6d51da1-a6c7-4060-8da8-74b8b56e949d\") " Oct 30 20:15:21.031372 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:21.031224 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6d51da1-a6c7-4060-8da8-74b8b56e949d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6d51da1-a6c7-4060-8da8-74b8b56e949d-istio-envoy\") pod \"c6d51da1-a6c7-4060-8da8-74b8b56e949d\" (UID: \"c6d51da1-a6c7-4060-8da8-74b8b56e949d\") " Oct 30 20:15:21.031372 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:21.031269 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6d51da1-a6c7-4060-8da8-74b8b56e949d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6d51da1-a6c7-4060-8da8-74b8b56e949d-workload-socket\") pod \"c6d51da1-a6c7-4060-8da8-74b8b56e949d\" (UID: \"c6d51da1-a6c7-4060-8da8-74b8b56e949d\") " Oct 30 20:15:21.031372 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:21.031294 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6d51da1-a6c7-4060-8da8-74b8b56e949d-kube-api-access-gq5ft\" (UniqueName: \"kubernetes.io/projected/c6d51da1-a6c7-4060-8da8-74b8b56e949d-kube-api-access-gq5ft\") pod \"c6d51da1-a6c7-4060-8da8-74b8b56e949d\" (UID: \"c6d51da1-a6c7-4060-8da8-74b8b56e949d\") " Oct 30 20:15:21.031372 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:21.031344 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c6d51da1-a6c7-4060-8da8-74b8b56e949d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6d51da1-a6c7-4060-8da8-74b8b56e949d-istio-podinfo\") pod \"c6d51da1-a6c7-4060-8da8-74b8b56e949d\" (UID: \"c6d51da1-a6c7-4060-8da8-74b8b56e949d\") " Oct 30 20:15:21.031654 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:21.031389 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6d51da1-a6c7-4060-8da8-74b8b56e949d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6d51da1-a6c7-4060-8da8-74b8b56e949d-istiod-ca-cert\") pod \"c6d51da1-a6c7-4060-8da8-74b8b56e949d\" (UID: \"c6d51da1-a6c7-4060-8da8-74b8b56e949d\") " Oct 30 20:15:21.031654 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:21.031452 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6d51da1-a6c7-4060-8da8-74b8b56e949d-istio-data" pod "c6d51da1-a6c7-4060-8da8-74b8b56e949d" (UID: "c6d51da1-a6c7-4060-8da8-74b8b56e949d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:21.031654 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:21.031463 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6d51da1-a6c7-4060-8da8-74b8b56e949d-credential-socket" pod "c6d51da1-a6c7-4060-8da8-74b8b56e949d" (UID: "c6d51da1-a6c7-4060-8da8-74b8b56e949d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:21.031654 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:21.031465 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6d51da1-a6c7-4060-8da8-74b8b56e949d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6d51da1-a6c7-4060-8da8-74b8b56e949d-workload-certs\") pod \"c6d51da1-a6c7-4060-8da8-74b8b56e949d\" (UID: \"c6d51da1-a6c7-4060-8da8-74b8b56e949d\") " Oct 30 20:15:21.031654 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:21.031594 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6d51da1-a6c7-4060-8da8-74b8b56e949d-istio-token\" (UniqueName: \"kubernetes.io/projected/c6d51da1-a6c7-4060-8da8-74b8b56e949d-istio-token\") pod \"c6d51da1-a6c7-4060-8da8-74b8b56e949d\" (UID: \"c6d51da1-a6c7-4060-8da8-74b8b56e949d\") " Oct 30 20:15:21.031901 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:21.031696 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6d51da1-a6c7-4060-8da8-74b8b56e949d-workload-certs" pod "c6d51da1-a6c7-4060-8da8-74b8b56e949d" (UID: "c6d51da1-a6c7-4060-8da8-74b8b56e949d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:21.031901 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:21.031710 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6d51da1-a6c7-4060-8da8-74b8b56e949d-workload-socket" pod "c6d51da1-a6c7-4060-8da8-74b8b56e949d" (UID: "c6d51da1-a6c7-4060-8da8-74b8b56e949d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:21.032147 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:21.032068 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6d51da1-a6c7-4060-8da8-74b8b56e949d-credential-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:21.032147 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:21.032094 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6d51da1-a6c7-4060-8da8-74b8b56e949d-istio-data\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:21.032147 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:21.032109 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6d51da1-a6c7-4060-8da8-74b8b56e949d-workload-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:21.032147 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:21.032124 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6d51da1-a6c7-4060-8da8-74b8b56e949d-workload-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:21.032147 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:21.032142 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6d51da1-a6c7-4060-8da8-74b8b56e949d-istiod-ca-cert" pod "c6d51da1-a6c7-4060-8da8-74b8b56e949d" (UID: "c6d51da1-a6c7-4060-8da8-74b8b56e949d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:21.033762 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:21.033740 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6d51da1-a6c7-4060-8da8-74b8b56e949d-kube-api-access-gq5ft" pod "c6d51da1-a6c7-4060-8da8-74b8b56e949d" (UID: "c6d51da1-a6c7-4060-8da8-74b8b56e949d"). InnerVolumeSpecName "kube-api-access-gq5ft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:21.034076 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:21.034049 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6d51da1-a6c7-4060-8da8-74b8b56e949d-istio-envoy" pod "c6d51da1-a6c7-4060-8da8-74b8b56e949d" (UID: "c6d51da1-a6c7-4060-8da8-74b8b56e949d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:21.034159 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:21.034089 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c6d51da1-a6c7-4060-8da8-74b8b56e949d-istio-podinfo" pod "c6d51da1-a6c7-4060-8da8-74b8b56e949d" (UID: "c6d51da1-a6c7-4060-8da8-74b8b56e949d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:15:21.034197 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:21.034156 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6d51da1-a6c7-4060-8da8-74b8b56e949d-istio-token" pod "c6d51da1-a6c7-4060-8da8-74b8b56e949d" (UID: "c6d51da1-a6c7-4060-8da8-74b8b56e949d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:21.133644 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:21.133603 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6d51da1-a6c7-4060-8da8-74b8b56e949d-istio-envoy\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:21.133644 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:21.133645 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq5ft\" (UniqueName: \"kubernetes.io/projected/c6d51da1-a6c7-4060-8da8-74b8b56e949d-kube-api-access-gq5ft\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:21.133860 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:21.133661 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6d51da1-a6c7-4060-8da8-74b8b56e949d-istio-podinfo\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:21.133860 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:21.133674 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6d51da1-a6c7-4060-8da8-74b8b56e949d-istiod-ca-cert\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:21.133860 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:21.133690 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6d51da1-a6c7-4060-8da8-74b8b56e949d-istio-token\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:21.197657 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:21.197624 4292 generic.go:356] "Generic (PLEG): container finished" podID="c6d51da1-a6c7-4060-8da8-74b8b56e949d" containerID="84b12bffa6d75a2e202ac3503c8a6b58f309b6e7334c3e86114db9acde76dc6d" exitCode=0 Oct 30 20:15:21.197829 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:21.197704 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5" Oct 30 20:15:21.197829 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:21.197722 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5" event={"ID":"c6d51da1-a6c7-4060-8da8-74b8b56e949d","Type":"ContainerDied","Data":"84b12bffa6d75a2e202ac3503c8a6b58f309b6e7334c3e86114db9acde76dc6d"} Oct 30 20:15:21.197829 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:21.197789 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5" event={"ID":"c6d51da1-a6c7-4060-8da8-74b8b56e949d","Type":"ContainerDied","Data":"ae0cdf1009986e6339b64ce248e1a738931dc9d60ed3f9bb5755f4c9776aa811"} Oct 30 20:15:21.197829 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:21.197828 4292 scope.go:122] "RemoveContainer" containerID="84b12bffa6d75a2e202ac3503c8a6b58f309b6e7334c3e86114db9acde76dc6d" Oct 30 20:15:21.209510 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:21.209434 4292 scope.go:122] "RemoveContainer" containerID="7306e784265f78402f6c9f9a644106ec8cc339a737ad42bf01206674024e8ca2" Oct 30 20:15:21.219098 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:21.219072 4292 scope.go:122] "RemoveContainer" containerID="84b12bffa6d75a2e202ac3503c8a6b58f309b6e7334c3e86114db9acde76dc6d" Oct 30 20:15:21.219400 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:15:21.219382 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84b12bffa6d75a2e202ac3503c8a6b58f309b6e7334c3e86114db9acde76dc6d\": container with ID starting with 84b12bffa6d75a2e202ac3503c8a6b58f309b6e7334c3e86114db9acde76dc6d not found: ID does not exist" containerID="84b12bffa6d75a2e202ac3503c8a6b58f309b6e7334c3e86114db9acde76dc6d" Oct 30 20:15:21.219466 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:21.219406 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84b12bffa6d75a2e202ac3503c8a6b58f309b6e7334c3e86114db9acde76dc6d"} err="failed to get container status \"84b12bffa6d75a2e202ac3503c8a6b58f309b6e7334c3e86114db9acde76dc6d\": rpc error: code = NotFound desc = could not find container \"84b12bffa6d75a2e202ac3503c8a6b58f309b6e7334c3e86114db9acde76dc6d\": container with ID starting with 84b12bffa6d75a2e202ac3503c8a6b58f309b6e7334c3e86114db9acde76dc6d not found: ID does not exist" Oct 30 20:15:21.219466 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:21.219425 4292 scope.go:122] "RemoveContainer" containerID="7306e784265f78402f6c9f9a644106ec8cc339a737ad42bf01206674024e8ca2" Oct 30 20:15:21.219708 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:15:21.219683 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7306e784265f78402f6c9f9a644106ec8cc339a737ad42bf01206674024e8ca2\": container with ID starting with 7306e784265f78402f6c9f9a644106ec8cc339a737ad42bf01206674024e8ca2 not found: ID does not exist" containerID="7306e784265f78402f6c9f9a644106ec8cc339a737ad42bf01206674024e8ca2" Oct 30 20:15:21.219762 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:21.219719 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7306e784265f78402f6c9f9a644106ec8cc339a737ad42bf01206674024e8ca2"} err="failed to get container status \"7306e784265f78402f6c9f9a644106ec8cc339a737ad42bf01206674024e8ca2\": rpc error: code = NotFound desc = could not find container \"7306e784265f78402f6c9f9a644106ec8cc339a737ad42bf01206674024e8ca2\": container with ID starting with 7306e784265f78402f6c9f9a644106ec8cc339a737ad42bf01206674024e8ca2 not found: ID does not exist" Oct 30 20:15:21.372647 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:21.372569 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5"] Oct 30 20:15:21.377824 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:21.377800 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-04679c-openshift-default-7b7b8b77bf-thkf5"] Oct 30 20:15:21.765102 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:21.765027 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr" Oct 30 20:15:21.765519 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:21.765289 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr" Oct 30 20:15:21.765969 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:21.765954 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr" Oct 30 20:15:22.582722 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:22.582688 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6d51da1-a6c7-4060-8da8-74b8b56e949d" path="/var/lib/kubelet/pods/c6d51da1-a6c7-4060-8da8-74b8b56e949d/volumes" Oct 30 20:15:27.084031 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:27.083992 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-389c4e/praxis-route-389c4e-kserve-7788b879cc-585s7" podUID="55a33bbd-4133-455e-8757-6cd6c6e229c2" containerName="main" containerID="cri-o://5c547e0107ea67ba56327332f496f9074281f01cfcb5721e43a46c7ce1a9c51b" gracePeriod=45 Oct 30 20:15:27.238874 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:27.238842 4292 generic.go:356] "Generic (PLEG): container finished" podID="55a33bbd-4133-455e-8757-6cd6c6e229c2" containerID="5c547e0107ea67ba56327332f496f9074281f01cfcb5721e43a46c7ce1a9c51b" exitCode=0 Oct 30 20:15:27.239027 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:27.238892 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-389c4e/praxis-route-389c4e-kserve-7788b879cc-585s7" event={"ID":"55a33bbd-4133-455e-8757-6cd6c6e229c2","Type":"ContainerDied","Data":"5c547e0107ea67ba56327332f496f9074281f01cfcb5721e43a46c7ce1a9c51b"} Oct 30 20:15:27.340547 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:27.340492 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-389c4e/praxis-route-389c4e-kserve-7788b879cc-585s7" Oct 30 20:15:27.394666 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:27.394636 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55a33bbd-4133-455e-8757-6cd6c6e229c2-home\" (UniqueName: \"kubernetes.io/empty-dir/55a33bbd-4133-455e-8757-6cd6c6e229c2-home\") pod \"55a33bbd-4133-455e-8757-6cd6c6e229c2\" (UID: \"55a33bbd-4133-455e-8757-6cd6c6e229c2\") " Oct 30 20:15:27.394830 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:27.394706 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55a33bbd-4133-455e-8757-6cd6c6e229c2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/55a33bbd-4133-455e-8757-6cd6c6e229c2-dshm\") pod \"55a33bbd-4133-455e-8757-6cd6c6e229c2\" (UID: \"55a33bbd-4133-455e-8757-6cd6c6e229c2\") " Oct 30 20:15:27.394830 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:27.394731 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55a33bbd-4133-455e-8757-6cd6c6e229c2-tls-certs\" (UniqueName: \"kubernetes.io/secret/55a33bbd-4133-455e-8757-6cd6c6e229c2-tls-certs\") pod \"55a33bbd-4133-455e-8757-6cd6c6e229c2\" (UID: \"55a33bbd-4133-455e-8757-6cd6c6e229c2\") " Oct 30 20:15:27.394830 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:27.394758 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55a33bbd-4133-455e-8757-6cd6c6e229c2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55a33bbd-4133-455e-8757-6cd6c6e229c2-tmp-dir\") pod \"55a33bbd-4133-455e-8757-6cd6c6e229c2\" (UID: \"55a33bbd-4133-455e-8757-6cd6c6e229c2\") " Oct 30 20:15:27.394830 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:27.394809 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55a33bbd-4133-455e-8757-6cd6c6e229c2-kube-api-access-76n7m\" (UniqueName: \"kubernetes.io/projected/55a33bbd-4133-455e-8757-6cd6c6e229c2-kube-api-access-76n7m\") pod \"55a33bbd-4133-455e-8757-6cd6c6e229c2\" (UID: \"55a33bbd-4133-455e-8757-6cd6c6e229c2\") " Oct 30 20:15:27.395046 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:27.394858 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55a33bbd-4133-455e-8757-6cd6c6e229c2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55a33bbd-4133-455e-8757-6cd6c6e229c2-model-cache\") pod \"55a33bbd-4133-455e-8757-6cd6c6e229c2\" (UID: \"55a33bbd-4133-455e-8757-6cd6c6e229c2\") " Oct 30 20:15:27.395046 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:27.394934 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55a33bbd-4133-455e-8757-6cd6c6e229c2-home" pod "55a33bbd-4133-455e-8757-6cd6c6e229c2" (UID: "55a33bbd-4133-455e-8757-6cd6c6e229c2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:27.395146 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:27.395096 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55a33bbd-4133-455e-8757-6cd6c6e229c2-tmp-dir" pod "55a33bbd-4133-455e-8757-6cd6c6e229c2" (UID: "55a33bbd-4133-455e-8757-6cd6c6e229c2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:27.395192 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:27.395156 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55a33bbd-4133-455e-8757-6cd6c6e229c2-model-cache" pod "55a33bbd-4133-455e-8757-6cd6c6e229c2" (UID: "55a33bbd-4133-455e-8757-6cd6c6e229c2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:27.395256 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:27.395241 4292 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55a33bbd-4133-455e-8757-6cd6c6e229c2-model-cache\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:27.395308 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:27.395253 4292 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55a33bbd-4133-455e-8757-6cd6c6e229c2-home\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:27.395308 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:27.395266 4292 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55a33bbd-4133-455e-8757-6cd6c6e229c2-tmp-dir\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:27.397034 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:27.397011 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55a33bbd-4133-455e-8757-6cd6c6e229c2-dshm" pod "55a33bbd-4133-455e-8757-6cd6c6e229c2" (UID: "55a33bbd-4133-455e-8757-6cd6c6e229c2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:27.397109 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:27.397014 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55a33bbd-4133-455e-8757-6cd6c6e229c2-tls-certs" pod "55a33bbd-4133-455e-8757-6cd6c6e229c2" (UID: "55a33bbd-4133-455e-8757-6cd6c6e229c2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:15:27.397109 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:27.397014 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55a33bbd-4133-455e-8757-6cd6c6e229c2-kube-api-access-76n7m" pod "55a33bbd-4133-455e-8757-6cd6c6e229c2" (UID: "55a33bbd-4133-455e-8757-6cd6c6e229c2"). InnerVolumeSpecName "kube-api-access-76n7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:27.496669 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:27.496641 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76n7m\" (UniqueName: \"kubernetes.io/projected/55a33bbd-4133-455e-8757-6cd6c6e229c2-kube-api-access-76n7m\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:27.496669 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:27.496665 4292 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55a33bbd-4133-455e-8757-6cd6c6e229c2-dshm\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:27.496669 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:27.496676 4292 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55a33bbd-4133-455e-8757-6cd6c6e229c2-tls-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:28.247970 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:28.247942 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-389c4e/praxis-route-389c4e-kserve-7788b879cc-585s7" Oct 30 20:15:28.248388 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:28.247986 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-389c4e/praxis-route-389c4e-kserve-7788b879cc-585s7" event={"ID":"55a33bbd-4133-455e-8757-6cd6c6e229c2","Type":"ContainerDied","Data":"9ae48489e1704d6cf93358b2227ee4c632e2f02a86da3be00b2078f009d332a5"} Oct 30 20:15:28.248388 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:28.248035 4292 scope.go:122] "RemoveContainer" containerID="5c547e0107ea67ba56327332f496f9074281f01cfcb5721e43a46c7ce1a9c51b" Oct 30 20:15:28.417082 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:28.417049 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-389c4e/praxis-route-389c4e-kserve-7788b879cc-585s7"] Oct 30 20:15:28.467482 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:28.467454 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-389c4e/praxis-route-389c4e-kserve-7788b879cc-585s7"] Oct 30 20:15:28.582263 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:28.582234 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55a33bbd-4133-455e-8757-6cd6c6e229c2" path="/var/lib/kubelet/pods/55a33bbd-4133-455e-8757-6cd6c6e229c2/volumes" Oct 30 20:15:37.342448 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:37.342419 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89eb1db5-hhnmg"] Oct 30 20:15:37.342952 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:37.342828 4292 cpu_manager.go:414] "containerMap: removing container" podUID="55a33bbd-4133-455e-8757-6cd6c6e229c2" containerName="main" Oct 30 20:15:37.342952 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:37.342840 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55a33bbd-4133-455e-8757-6cd6c6e229c2" containerName="main" Oct 30 20:15:37.342952 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:37.342856 4292 cpu_manager.go:414] "containerMap: removing container" podUID="0a4c41bd-fc28-4552-be12-170b75585dae" containerName="cleanup" Oct 30 20:15:37.342952 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:37.342862 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a4c41bd-fc28-4552-be12-170b75585dae" containerName="cleanup" Oct 30 20:15:37.342952 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:37.342874 4292 cpu_manager.go:414] "containerMap: removing container" podUID="6c74471c-9ea5-486f-b083-efee6a92777d" containerName="cleanup" Oct 30 20:15:37.342952 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:37.342878 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c74471c-9ea5-486f-b083-efee6a92777d" containerName="cleanup" Oct 30 20:15:37.342952 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:37.342887 4292 cpu_manager.go:414] "containerMap: removing container" podUID="c6d51da1-a6c7-4060-8da8-74b8b56e949d" containerName="istio-proxy" Oct 30 20:15:37.342952 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:37.342892 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6d51da1-a6c7-4060-8da8-74b8b56e949d" containerName="istio-proxy" Oct 30 20:15:37.342952 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:37.342899 4292 cpu_manager.go:414] "containerMap: removing container" podUID="e455047b-0948-4538-a504-9e0a3324b00c" containerName="cleanup" Oct 30 20:15:37.342952 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:37.342903 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e455047b-0948-4538-a504-9e0a3324b00c" containerName="cleanup" Oct 30 20:15:37.342952 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:37.342911 4292 cpu_manager.go:414] "containerMap: removing container" podUID="c6d51da1-a6c7-4060-8da8-74b8b56e949d" containerName="istio-proxy" Oct 30 20:15:37.342952 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:37.342915 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6d51da1-a6c7-4060-8da8-74b8b56e949d" containerName="istio-proxy" Oct 30 20:15:37.342952 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:37.342921 4292 cpu_manager.go:414] "containerMap: removing container" podUID="6c74471c-9ea5-486f-b083-efee6a92777d" containerName="cleanup" Oct 30 20:15:37.342952 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:37.342925 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c74471c-9ea5-486f-b083-efee6a92777d" containerName="cleanup" Oct 30 20:15:37.342952 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:37.342932 4292 cpu_manager.go:414] "containerMap: removing container" podUID="e455047b-0948-4538-a504-9e0a3324b00c" containerName="cleanup" Oct 30 20:15:37.342952 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:37.342936 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e455047b-0948-4538-a504-9e0a3324b00c" containerName="cleanup" Oct 30 20:15:37.342952 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:37.342941 4292 cpu_manager.go:414] "containerMap: removing container" podUID="6c74471c-9ea5-486f-b083-efee6a92777d" containerName="cleanup" Oct 30 20:15:37.342952 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:37.342945 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c74471c-9ea5-486f-b083-efee6a92777d" containerName="cleanup" Oct 30 20:15:37.343520 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:37.342964 4292 cpu_manager.go:414] "containerMap: removing container" podUID="0a4c41bd-fc28-4552-be12-170b75585dae" containerName="cleanup" Oct 30 20:15:37.343520 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:37.342968 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a4c41bd-fc28-4552-be12-170b75585dae" containerName="cleanup" Oct 30 20:15:37.343520 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:37.343030 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a4c41bd-fc28-4552-be12-170b75585dae" containerName="cleanup" Oct 30 20:15:37.343520 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:37.343037 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c74471c-9ea5-486f-b083-efee6a92777d" containerName="cleanup" Oct 30 20:15:37.343520 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:37.343043 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6d51da1-a6c7-4060-8da8-74b8b56e949d" containerName="istio-proxy" Oct 30 20:15:37.343520 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:37.343048 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="e455047b-0948-4538-a504-9e0a3324b00c" containerName="cleanup" Oct 30 20:15:37.343520 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:37.343053 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6d51da1-a6c7-4060-8da8-74b8b56e949d" containerName="istio-proxy" Oct 30 20:15:37.343520 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:37.343057 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c74471c-9ea5-486f-b083-efee6a92777d" containerName="cleanup" Oct 30 20:15:37.343520 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:37.343064 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a4c41bd-fc28-4552-be12-170b75585dae" containerName="cleanup" Oct 30 20:15:37.343520 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:37.343069 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="e455047b-0948-4538-a504-9e0a3324b00c" containerName="cleanup" Oct 30 20:15:37.343520 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:37.343073 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="55a33bbd-4133-455e-8757-6cd6c6e229c2" containerName="main" Oct 30 20:15:37.347470 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:37.347454 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89eb1db5-hhnmg" Oct 30 20:15:37.396819 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:37.396788 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c36eb17f-aab3-467e-b983-daf894b5226f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89eb1db5-hhnmg\" (UID: \"c36eb17f-aab3-467e-b983-daf894b5226f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89eb1db5-hhnmg" Oct 30 20:15:37.396966 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:37.396848 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm9nm\" (UniqueName: \"kubernetes.io/projected/c36eb17f-aab3-467e-b983-daf894b5226f-kube-api-access-tm9nm\") pod \"maas-api-revoke-sub-89eb1db5-hhnmg\" (UID: \"c36eb17f-aab3-467e-b983-daf894b5226f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89eb1db5-hhnmg" Oct 30 20:15:37.397012 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:37.396986 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c36eb17f-aab3-467e-b983-daf894b5226f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89eb1db5-hhnmg\" (UID: \"c36eb17f-aab3-467e-b983-daf894b5226f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89eb1db5-hhnmg" Oct 30 20:15:37.402785 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:37.402759 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89eb1db5-hhnmg"] Oct 30 20:15:37.450019 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:37.449989 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:15:37.450144 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:37.450086 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\"" Oct 30 20:15:37.498552 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:37.498509 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c36eb17f-aab3-467e-b983-daf894b5226f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89eb1db5-hhnmg\" (UID: \"c36eb17f-aab3-467e-b983-daf894b5226f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89eb1db5-hhnmg" Oct 30 20:15:37.498696 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:37.498568 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm9nm\" (UniqueName: \"kubernetes.io/projected/c36eb17f-aab3-467e-b983-daf894b5226f-kube-api-access-tm9nm\") pod \"maas-api-revoke-sub-89eb1db5-hhnmg\" (UID: \"c36eb17f-aab3-467e-b983-daf894b5226f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89eb1db5-hhnmg" Oct 30 20:15:37.498696 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:37.498633 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c36eb17f-aab3-467e-b983-daf894b5226f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89eb1db5-hhnmg\" (UID: \"c36eb17f-aab3-467e-b983-daf894b5226f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89eb1db5-hhnmg" Oct 30 20:15:37.499180 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:37.499161 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c36eb17f-aab3-467e-b983-daf894b5226f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89eb1db5-hhnmg\" (UID: \"c36eb17f-aab3-467e-b983-daf894b5226f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89eb1db5-hhnmg" Oct 30 20:15:37.500933 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:37.500911 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c36eb17f-aab3-467e-b983-daf894b5226f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89eb1db5-hhnmg\" (UID: \"c36eb17f-aab3-467e-b983-daf894b5226f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89eb1db5-hhnmg" Oct 30 20:15:37.561200 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:37.561177 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm9nm\" (UniqueName: \"kubernetes.io/projected/c36eb17f-aab3-467e-b983-daf894b5226f-kube-api-access-tm9nm\") pod \"maas-api-revoke-sub-89eb1db5-hhnmg\" (UID: \"c36eb17f-aab3-467e-b983-daf894b5226f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89eb1db5-hhnmg" Oct 30 20:15:37.660135 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:37.660061 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89eb1db5-hhnmg" Oct 30 20:15:38.029876 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:15:38.029842 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc36eb17f_aab3_467e_b983_daf894b5226f.slice/crio-2a526ede854131b8cd227043fd888dd7bd8ae01d9f31d13a40803735ef7e1d73 WatchSource:0}: Error finding container 2a526ede854131b8cd227043fd888dd7bd8ae01d9f31d13a40803735ef7e1d73: Status 404 returned error can't find the container with id 2a526ede854131b8cd227043fd888dd7bd8ae01d9f31d13a40803735ef7e1d73 Oct 30 20:15:38.033211 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:38.033189 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89eb1db5-hhnmg"] Oct 30 20:15:38.311236 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:38.311151 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89eb1db5-hhnmg" event={"ID":"c36eb17f-aab3-467e-b983-daf894b5226f","Type":"ContainerStarted","Data":"2a526ede854131b8cd227043fd888dd7bd8ae01d9f31d13a40803735ef7e1d73"} Oct 30 20:15:39.318170 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:39.318138 4292 generic.go:356] "Generic (PLEG): container finished" podID="c36eb17f-aab3-467e-b983-daf894b5226f" containerID="35fa25dcfa29c7b7fc43847f3a1c4778221cde63929a198412d58bcb15d89919" exitCode=0 Oct 30 20:15:39.318581 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:39.318226 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89eb1db5-hhnmg" event={"ID":"c36eb17f-aab3-467e-b983-daf894b5226f","Type":"ContainerDied","Data":"35fa25dcfa29c7b7fc43847f3a1c4778221cde63929a198412d58bcb15d89919"} Oct 30 20:15:40.457177 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:40.457152 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89eb1db5-hhnmg" Oct 30 20:15:40.531495 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:40.531468 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c36eb17f-aab3-467e-b983-daf894b5226f-kube-api-access-tm9nm\" (UniqueName: \"kubernetes.io/projected/c36eb17f-aab3-467e-b983-daf894b5226f-kube-api-access-tm9nm\") pod \"c36eb17f-aab3-467e-b983-daf894b5226f\" (UID: \"c36eb17f-aab3-467e-b983-daf894b5226f\") " Oct 30 20:15:40.531626 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:40.531532 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c36eb17f-aab3-467e-b983-daf894b5226f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c36eb17f-aab3-467e-b983-daf894b5226f-maas-api-xks-ca\") pod \"c36eb17f-aab3-467e-b983-daf894b5226f\" (UID: \"c36eb17f-aab3-467e-b983-daf894b5226f\") " Oct 30 20:15:40.531678 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:40.531645 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c36eb17f-aab3-467e-b983-daf894b5226f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c36eb17f-aab3-467e-b983-daf894b5226f-maas-api-openshift-ca\") pod \"c36eb17f-aab3-467e-b983-daf894b5226f\" (UID: \"c36eb17f-aab3-467e-b983-daf894b5226f\") " Oct 30 20:15:40.532011 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:40.531985 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c36eb17f-aab3-467e-b983-daf894b5226f-maas-api-openshift-ca" pod "c36eb17f-aab3-467e-b983-daf894b5226f" (UID: "c36eb17f-aab3-467e-b983-daf894b5226f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:40.533522 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:40.533499 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c36eb17f-aab3-467e-b983-daf894b5226f-kube-api-access-tm9nm" pod "c36eb17f-aab3-467e-b983-daf894b5226f" (UID: "c36eb17f-aab3-467e-b983-daf894b5226f"). InnerVolumeSpecName "kube-api-access-tm9nm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:40.533611 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:40.533579 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c36eb17f-aab3-467e-b983-daf894b5226f-maas-api-xks-ca" pod "c36eb17f-aab3-467e-b983-daf894b5226f" (UID: "c36eb17f-aab3-467e-b983-daf894b5226f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:15:40.632915 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:40.632885 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tm9nm\" (UniqueName: \"kubernetes.io/projected/c36eb17f-aab3-467e-b983-daf894b5226f-kube-api-access-tm9nm\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:40.632915 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:40.632914 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c36eb17f-aab3-467e-b983-daf894b5226f-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:40.633145 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:40.632923 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c36eb17f-aab3-467e-b983-daf894b5226f-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:41.331999 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:41.331966 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89eb1db5-hhnmg" Oct 30 20:15:41.332197 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:41.332014 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89eb1db5-hhnmg" event={"ID":"c36eb17f-aab3-467e-b983-daf894b5226f","Type":"ContainerDied","Data":"2a526ede854131b8cd227043fd888dd7bd8ae01d9f31d13a40803735ef7e1d73"} Oct 30 20:15:41.332197 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:41.332061 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a526ede854131b8cd227043fd888dd7bd8ae01d9f31d13a40803735ef7e1d73" Oct 30 20:15:43.220459 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:15:43.220416 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:15:58.122304 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:58.122270 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49"] Oct 30 20:15:58.122804 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:58.122567 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49" podUID="ed835dd4-48ed-46cf-9180-d616d5fcec23" containerName="istio-proxy" containerID="cri-o://db8d85f2e8d793cc7e5b2414cb52537961457cab561346fea480117faf26da92" gracePeriod=30 Oct 30 20:15:58.128978 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:58.128952 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd"] Oct 30 20:15:58.129139 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:58.129119 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd" podUID="095f29a0-8906-4da7-89ab-95fa4f9dc1a7" containerName="istio-proxy" containerID="cri-o://62b7b5d5dc6af96b9cb548456d4ae9c59a0e27ddf2e47a8a86e82708ec218cdd" gracePeriod=30 Oct 30 20:15:58.993772 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:58.993737 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f53eb9-d7vlt"] Oct 30 20:15:58.994204 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:58.994190 4292 cpu_manager.go:414] "containerMap: removing container" podUID="e455047b-0948-4538-a504-9e0a3324b00c" containerName="cleanup" Oct 30 20:15:58.994269 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:58.994206 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e455047b-0948-4538-a504-9e0a3324b00c" containerName="cleanup" Oct 30 20:15:58.994269 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:58.994225 4292 cpu_manager.go:414] "containerMap: removing container" podUID="0a4c41bd-fc28-4552-be12-170b75585dae" containerName="cleanup" Oct 30 20:15:58.994269 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:58.994230 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a4c41bd-fc28-4552-be12-170b75585dae" containerName="cleanup" Oct 30 20:15:58.994269 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:58.994247 4292 cpu_manager.go:414] "containerMap: removing container" podUID="c36eb17f-aab3-467e-b983-daf894b5226f" containerName="revoke-keys" Oct 30 20:15:58.994269 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:58.994253 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c36eb17f-aab3-467e-b983-daf894b5226f" containerName="revoke-keys" Oct 30 20:15:58.994567 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:58.994320 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c74471c-9ea5-486f-b083-efee6a92777d" containerName="cleanup" Oct 30 20:15:58.994567 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:58.994333 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a4c41bd-fc28-4552-be12-170b75585dae" containerName="cleanup" Oct 30 20:15:58.994567 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:58.994342 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="c36eb17f-aab3-467e-b983-daf894b5226f" containerName="revoke-keys" Oct 30 20:15:58.994567 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:58.994375 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="e455047b-0948-4538-a504-9e0a3324b00c" containerName="cleanup" Oct 30 20:15:59.000328 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:59.000309 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f53eb9-d7vlt" Oct 30 20:15:59.051797 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:59.051765 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f53eb9-d7vlt"] Oct 30 20:15:59.077292 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:59.077256 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:15:59.077515 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:59.077404 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\"" Oct 30 20:15:59.162834 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:59.162798 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bbe9c82-995f-4598-a2b6-28b6495f2921-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-f53eb9-d7vlt\" (UID: \"7bbe9c82-995f-4598-a2b6-28b6495f2921\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f53eb9-d7vlt" Oct 30 20:15:59.163190 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:59.162870 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6l5k\" (UniqueName: \"kubernetes.io/projected/7bbe9c82-995f-4598-a2b6-28b6495f2921-kube-api-access-d6l5k\") pod \"maas-api-revoke-keys-e2e-praxis-b-f53eb9-d7vlt\" (UID: \"7bbe9c82-995f-4598-a2b6-28b6495f2921\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f53eb9-d7vlt" Oct 30 20:15:59.163190 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:59.162937 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bbe9c82-995f-4598-a2b6-28b6495f2921-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-f53eb9-d7vlt\" (UID: \"7bbe9c82-995f-4598-a2b6-28b6495f2921\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f53eb9-d7vlt" Oct 30 20:15:59.263622 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:59.263516 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bbe9c82-995f-4598-a2b6-28b6495f2921-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-f53eb9-d7vlt\" (UID: \"7bbe9c82-995f-4598-a2b6-28b6495f2921\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f53eb9-d7vlt" Oct 30 20:15:59.263804 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:59.263663 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bbe9c82-995f-4598-a2b6-28b6495f2921-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-f53eb9-d7vlt\" (UID: \"7bbe9c82-995f-4598-a2b6-28b6495f2921\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f53eb9-d7vlt" Oct 30 20:15:59.263879 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:59.263808 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6l5k\" (UniqueName: \"kubernetes.io/projected/7bbe9c82-995f-4598-a2b6-28b6495f2921-kube-api-access-d6l5k\") pod \"maas-api-revoke-keys-e2e-praxis-b-f53eb9-d7vlt\" (UID: \"7bbe9c82-995f-4598-a2b6-28b6495f2921\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f53eb9-d7vlt" Oct 30 20:15:59.264189 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:59.264166 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bbe9c82-995f-4598-a2b6-28b6495f2921-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-f53eb9-d7vlt\" (UID: \"7bbe9c82-995f-4598-a2b6-28b6495f2921\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f53eb9-d7vlt" Oct 30 20:15:59.266052 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:59.266030 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bbe9c82-995f-4598-a2b6-28b6495f2921-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-f53eb9-d7vlt\" (UID: \"7bbe9c82-995f-4598-a2b6-28b6495f2921\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f53eb9-d7vlt" Oct 30 20:15:59.326009 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:59.325985 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6l5k\" (UniqueName: \"kubernetes.io/projected/7bbe9c82-995f-4598-a2b6-28b6495f2921-kube-api-access-d6l5k\") pod \"maas-api-revoke-keys-e2e-praxis-b-f53eb9-d7vlt\" (UID: \"7bbe9c82-995f-4598-a2b6-28b6495f2921\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f53eb9-d7vlt" Oct 30 20:15:59.613058 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:59.613029 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f53eb9-d7vlt" Oct 30 20:15:59.984724 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:15:59.984695 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bbe9c82_995f_4598_a2b6_28b6495f2921.slice/crio-b79be567560354757d3766dc48394958a2f27bbafe5246609cbdaca95b9716d4 WatchSource:0}: Error finding container b79be567560354757d3766dc48394958a2f27bbafe5246609cbdaca95b9716d4: Status 404 returned error can't find the container with id b79be567560354757d3766dc48394958a2f27bbafe5246609cbdaca95b9716d4 Oct 30 20:15:59.999618 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:15:59.992314 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f53eb9-d7vlt"] Oct 30 20:16:00.489962 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:00.489920 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f53eb9-d7vlt" event={"ID":"7bbe9c82-995f-4598-a2b6-28b6495f2921","Type":"ContainerStarted","Data":"3a3fd19a9d82513c0163be0949e58010e6ce6a481721e1b11970092b42dda768"} Oct 30 20:16:00.490435 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:00.489989 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f53eb9-d7vlt" event={"ID":"7bbe9c82-995f-4598-a2b6-28b6495f2921","Type":"ContainerStarted","Data":"b79be567560354757d3766dc48394958a2f27bbafe5246609cbdaca95b9716d4"} Oct 30 20:16:00.610933 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:00.610877 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f53eb9-d7vlt" podStartSLOduration=2.610863174 podStartE2EDuration="2.610863174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:00.608845039 +0000 UTC m=+3064.127160519" watchObservedRunningTime="2026-10-30 20:16:00.610863174 +0000 UTC m=+3064.129178658" Oct 30 20:16:01.498599 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:01.498551 4292 generic.go:356] "Generic (PLEG): container finished" podID="7bbe9c82-995f-4598-a2b6-28b6495f2921" containerID="3a3fd19a9d82513c0163be0949e58010e6ce6a481721e1b11970092b42dda768" exitCode=0 Oct 30 20:16:01.499105 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:01.498636 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f53eb9-d7vlt" event={"ID":"7bbe9c82-995f-4598-a2b6-28b6495f2921","Type":"ContainerDied","Data":"3a3fd19a9d82513c0163be0949e58010e6ce6a481721e1b11970092b42dda768"} Oct 30 20:16:02.648873 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:02.648849 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f53eb9-d7vlt" Oct 30 20:16:02.720081 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:02.720050 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bbe9c82-995f-4598-a2b6-28b6495f2921-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bbe9c82-995f-4598-a2b6-28b6495f2921-maas-api-xks-ca\") pod \"7bbe9c82-995f-4598-a2b6-28b6495f2921\" (UID: \"7bbe9c82-995f-4598-a2b6-28b6495f2921\") " Oct 30 20:16:02.720213 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:02.720136 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bbe9c82-995f-4598-a2b6-28b6495f2921-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bbe9c82-995f-4598-a2b6-28b6495f2921-maas-api-openshift-ca\") pod \"7bbe9c82-995f-4598-a2b6-28b6495f2921\" (UID: \"7bbe9c82-995f-4598-a2b6-28b6495f2921\") " Oct 30 20:16:02.720213 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:02.720165 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bbe9c82-995f-4598-a2b6-28b6495f2921-kube-api-access-d6l5k\" (UniqueName: \"kubernetes.io/projected/7bbe9c82-995f-4598-a2b6-28b6495f2921-kube-api-access-d6l5k\") pod \"7bbe9c82-995f-4598-a2b6-28b6495f2921\" (UID: \"7bbe9c82-995f-4598-a2b6-28b6495f2921\") " Oct 30 20:16:02.720481 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:02.720459 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bbe9c82-995f-4598-a2b6-28b6495f2921-maas-api-openshift-ca" pod "7bbe9c82-995f-4598-a2b6-28b6495f2921" (UID: "7bbe9c82-995f-4598-a2b6-28b6495f2921"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:16:02.722231 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:02.722199 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bbe9c82-995f-4598-a2b6-28b6495f2921-maas-api-xks-ca" pod "7bbe9c82-995f-4598-a2b6-28b6495f2921" (UID: "7bbe9c82-995f-4598-a2b6-28b6495f2921"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:16:02.722231 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:02.722206 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bbe9c82-995f-4598-a2b6-28b6495f2921-kube-api-access-d6l5k" pod "7bbe9c82-995f-4598-a2b6-28b6495f2921" (UID: "7bbe9c82-995f-4598-a2b6-28b6495f2921"). InnerVolumeSpecName "kube-api-access-d6l5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:16:02.821672 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:02.821638 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bbe9c82-995f-4598-a2b6-28b6495f2921-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:02.821672 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:02.821669 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bbe9c82-995f-4598-a2b6-28b6495f2921-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:02.821672 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:02.821679 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6l5k\" (UniqueName: \"kubernetes.io/projected/7bbe9c82-995f-4598-a2b6-28b6495f2921-kube-api-access-d6l5k\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:03.367872 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.367838 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49" Oct 30 20:16:03.403209 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.403187 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd" Oct 30 20:16:03.428823 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.428797 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed835dd4-48ed-46cf-9180-d616d5fcec23-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed835dd4-48ed-46cf-9180-d616d5fcec23-workload-socket\") pod \"ed835dd4-48ed-46cf-9180-d616d5fcec23\" (UID: \"ed835dd4-48ed-46cf-9180-d616d5fcec23\") " Oct 30 20:16:03.428971 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.428869 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed835dd4-48ed-46cf-9180-d616d5fcec23-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed835dd4-48ed-46cf-9180-d616d5fcec23-workload-certs\") pod \"ed835dd4-48ed-46cf-9180-d616d5fcec23\" (UID: \"ed835dd4-48ed-46cf-9180-d616d5fcec23\") " Oct 30 20:16:03.428971 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.428901 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/095f29a0-8906-4da7-89ab-95fa4f9dc1a7-istio-token\" (UniqueName: \"kubernetes.io/projected/095f29a0-8906-4da7-89ab-95fa4f9dc1a7-istio-token\") pod \"095f29a0-8906-4da7-89ab-95fa4f9dc1a7\" (UID: \"095f29a0-8906-4da7-89ab-95fa4f9dc1a7\") " Oct 30 20:16:03.428971 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.428919 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/095f29a0-8906-4da7-89ab-95fa4f9dc1a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/095f29a0-8906-4da7-89ab-95fa4f9dc1a7-istio-podinfo\") pod \"095f29a0-8906-4da7-89ab-95fa4f9dc1a7\" (UID: \"095f29a0-8906-4da7-89ab-95fa4f9dc1a7\") " Oct 30 20:16:03.428971 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.428946 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed835dd4-48ed-46cf-9180-d616d5fcec23-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed835dd4-48ed-46cf-9180-d616d5fcec23-istiod-ca-cert\") pod \"ed835dd4-48ed-46cf-9180-d616d5fcec23\" (UID: \"ed835dd4-48ed-46cf-9180-d616d5fcec23\") " Oct 30 20:16:03.429185 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.428984 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/095f29a0-8906-4da7-89ab-95fa4f9dc1a7-kube-api-access-xtclw\" (UniqueName: \"kubernetes.io/projected/095f29a0-8906-4da7-89ab-95fa4f9dc1a7-kube-api-access-xtclw\") pod \"095f29a0-8906-4da7-89ab-95fa4f9dc1a7\" (UID: \"095f29a0-8906-4da7-89ab-95fa4f9dc1a7\") " Oct 30 20:16:03.429185 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.429024 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/095f29a0-8906-4da7-89ab-95fa4f9dc1a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/095f29a0-8906-4da7-89ab-95fa4f9dc1a7-istio-data\") pod \"095f29a0-8906-4da7-89ab-95fa4f9dc1a7\" (UID: \"095f29a0-8906-4da7-89ab-95fa4f9dc1a7\") " Oct 30 20:16:03.429185 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.429075 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/095f29a0-8906-4da7-89ab-95fa4f9dc1a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/095f29a0-8906-4da7-89ab-95fa4f9dc1a7-workload-socket\") pod \"095f29a0-8906-4da7-89ab-95fa4f9dc1a7\" (UID: \"095f29a0-8906-4da7-89ab-95fa4f9dc1a7\") " Oct 30 20:16:03.429185 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.429077 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed835dd4-48ed-46cf-9180-d616d5fcec23-workload-socket" pod "ed835dd4-48ed-46cf-9180-d616d5fcec23" (UID: "ed835dd4-48ed-46cf-9180-d616d5fcec23"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:03.429185 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.429105 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/095f29a0-8906-4da7-89ab-95fa4f9dc1a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/095f29a0-8906-4da7-89ab-95fa4f9dc1a7-credential-socket\") pod \"095f29a0-8906-4da7-89ab-95fa4f9dc1a7\" (UID: \"095f29a0-8906-4da7-89ab-95fa4f9dc1a7\") " Oct 30 20:16:03.429185 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.429130 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/095f29a0-8906-4da7-89ab-95fa4f9dc1a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/095f29a0-8906-4da7-89ab-95fa4f9dc1a7-workload-certs\") pod \"095f29a0-8906-4da7-89ab-95fa4f9dc1a7\" (UID: \"095f29a0-8906-4da7-89ab-95fa4f9dc1a7\") " Oct 30 20:16:03.429185 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.429172 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed835dd4-48ed-46cf-9180-d616d5fcec23-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed835dd4-48ed-46cf-9180-d616d5fcec23-istio-podinfo\") pod \"ed835dd4-48ed-46cf-9180-d616d5fcec23\" (UID: \"ed835dd4-48ed-46cf-9180-d616d5fcec23\") " Oct 30 20:16:03.429185 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.429175 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed835dd4-48ed-46cf-9180-d616d5fcec23-workload-certs" pod "ed835dd4-48ed-46cf-9180-d616d5fcec23" (UID: "ed835dd4-48ed-46cf-9180-d616d5fcec23"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:03.429633 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.429199 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed835dd4-48ed-46cf-9180-d616d5fcec23-istio-token\" (UniqueName: \"kubernetes.io/projected/ed835dd4-48ed-46cf-9180-d616d5fcec23-istio-token\") pod \"ed835dd4-48ed-46cf-9180-d616d5fcec23\" (UID: \"ed835dd4-48ed-46cf-9180-d616d5fcec23\") " Oct 30 20:16:03.429633 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.429245 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed835dd4-48ed-46cf-9180-d616d5fcec23-kube-api-access-62td5\" (UniqueName: \"kubernetes.io/projected/ed835dd4-48ed-46cf-9180-d616d5fcec23-kube-api-access-62td5\") pod \"ed835dd4-48ed-46cf-9180-d616d5fcec23\" (UID: \"ed835dd4-48ed-46cf-9180-d616d5fcec23\") " Oct 30 20:16:03.429633 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.429278 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed835dd4-48ed-46cf-9180-d616d5fcec23-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed835dd4-48ed-46cf-9180-d616d5fcec23-istio-data\") pod \"ed835dd4-48ed-46cf-9180-d616d5fcec23\" (UID: \"ed835dd4-48ed-46cf-9180-d616d5fcec23\") " Oct 30 20:16:03.429633 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.429304 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/095f29a0-8906-4da7-89ab-95fa4f9dc1a7-istio-data" pod "095f29a0-8906-4da7-89ab-95fa4f9dc1a7" (UID: "095f29a0-8906-4da7-89ab-95fa4f9dc1a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:03.429633 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.429323 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/095f29a0-8906-4da7-89ab-95fa4f9dc1a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/095f29a0-8906-4da7-89ab-95fa4f9dc1a7-istiod-ca-cert\") pod \"095f29a0-8906-4da7-89ab-95fa4f9dc1a7\" (UID: \"095f29a0-8906-4da7-89ab-95fa4f9dc1a7\") " Oct 30 20:16:03.429633 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.429447 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed835dd4-48ed-46cf-9180-d616d5fcec23-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed835dd4-48ed-46cf-9180-d616d5fcec23-istio-envoy\") pod \"ed835dd4-48ed-46cf-9180-d616d5fcec23\" (UID: \"ed835dd4-48ed-46cf-9180-d616d5fcec23\") " Oct 30 20:16:03.429633 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.429474 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed835dd4-48ed-46cf-9180-d616d5fcec23-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed835dd4-48ed-46cf-9180-d616d5fcec23-credential-socket\") pod \"ed835dd4-48ed-46cf-9180-d616d5fcec23\" (UID: \"ed835dd4-48ed-46cf-9180-d616d5fcec23\") " Oct 30 20:16:03.429633 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.429471 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/095f29a0-8906-4da7-89ab-95fa4f9dc1a7-workload-certs" pod "095f29a0-8906-4da7-89ab-95fa4f9dc1a7" (UID: "095f29a0-8906-4da7-89ab-95fa4f9dc1a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:03.429633 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.429572 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/095f29a0-8906-4da7-89ab-95fa4f9dc1a7-workload-socket" pod "095f29a0-8906-4da7-89ab-95fa4f9dc1a7" (UID: "095f29a0-8906-4da7-89ab-95fa4f9dc1a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:03.429633 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.429574 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/095f29a0-8906-4da7-89ab-95fa4f9dc1a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/095f29a0-8906-4da7-89ab-95fa4f9dc1a7-istio-envoy\") pod \"095f29a0-8906-4da7-89ab-95fa4f9dc1a7\" (UID: \"095f29a0-8906-4da7-89ab-95fa4f9dc1a7\") " Oct 30 20:16:03.429633 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.429605 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/095f29a0-8906-4da7-89ab-95fa4f9dc1a7-credential-socket" pod "095f29a0-8906-4da7-89ab-95fa4f9dc1a7" (UID: "095f29a0-8906-4da7-89ab-95fa4f9dc1a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:03.430259 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.429696 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed835dd4-48ed-46cf-9180-d616d5fcec23-credential-socket" pod "ed835dd4-48ed-46cf-9180-d616d5fcec23" (UID: "ed835dd4-48ed-46cf-9180-d616d5fcec23"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:03.430259 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.429896 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed835dd4-48ed-46cf-9180-d616d5fcec23-istiod-ca-cert" pod "ed835dd4-48ed-46cf-9180-d616d5fcec23" (UID: "ed835dd4-48ed-46cf-9180-d616d5fcec23"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:16:03.430259 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.429901 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed835dd4-48ed-46cf-9180-d616d5fcec23-istio-data" pod "ed835dd4-48ed-46cf-9180-d616d5fcec23" (UID: "ed835dd4-48ed-46cf-9180-d616d5fcec23"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:03.430259 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.430185 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed835dd4-48ed-46cf-9180-d616d5fcec23-workload-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:03.430259 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.430204 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/095f29a0-8906-4da7-89ab-95fa4f9dc1a7-istiod-ca-cert" pod "095f29a0-8906-4da7-89ab-95fa4f9dc1a7" (UID: "095f29a0-8906-4da7-89ab-95fa4f9dc1a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:16:03.430259 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.430209 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed835dd4-48ed-46cf-9180-d616d5fcec23-istiod-ca-cert\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:03.430259 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.430253 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/095f29a0-8906-4da7-89ab-95fa4f9dc1a7-istio-data\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:03.430729 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.430269 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/095f29a0-8906-4da7-89ab-95fa4f9dc1a7-workload-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:03.430729 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.430287 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/095f29a0-8906-4da7-89ab-95fa4f9dc1a7-credential-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:03.430729 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.430301 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/095f29a0-8906-4da7-89ab-95fa4f9dc1a7-workload-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:03.430729 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.430315 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed835dd4-48ed-46cf-9180-d616d5fcec23-istio-data\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:03.430729 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.430329 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed835dd4-48ed-46cf-9180-d616d5fcec23-credential-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:03.430729 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.430341 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed835dd4-48ed-46cf-9180-d616d5fcec23-workload-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:03.431732 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.431697 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/095f29a0-8906-4da7-89ab-95fa4f9dc1a7-istio-podinfo" pod "095f29a0-8906-4da7-89ab-95fa4f9dc1a7" (UID: "095f29a0-8906-4da7-89ab-95fa4f9dc1a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:16:03.432143 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.432105 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/095f29a0-8906-4da7-89ab-95fa4f9dc1a7-kube-api-access-xtclw" pod "095f29a0-8906-4da7-89ab-95fa4f9dc1a7" (UID: "095f29a0-8906-4da7-89ab-95fa4f9dc1a7"). InnerVolumeSpecName "kube-api-access-xtclw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:16:03.432313 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.432289 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/095f29a0-8906-4da7-89ab-95fa4f9dc1a7-istio-envoy" pod "095f29a0-8906-4da7-89ab-95fa4f9dc1a7" (UID: "095f29a0-8906-4da7-89ab-95fa4f9dc1a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:03.432417 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.432312 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed835dd4-48ed-46cf-9180-d616d5fcec23-kube-api-access-62td5" pod "ed835dd4-48ed-46cf-9180-d616d5fcec23" (UID: "ed835dd4-48ed-46cf-9180-d616d5fcec23"). InnerVolumeSpecName "kube-api-access-62td5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:16:03.432417 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.432340 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/095f29a0-8906-4da7-89ab-95fa4f9dc1a7-istio-token" pod "095f29a0-8906-4da7-89ab-95fa4f9dc1a7" (UID: "095f29a0-8906-4da7-89ab-95fa4f9dc1a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:16:03.432588 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.432573 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed835dd4-48ed-46cf-9180-d616d5fcec23-istio-token" pod "ed835dd4-48ed-46cf-9180-d616d5fcec23" (UID: "ed835dd4-48ed-46cf-9180-d616d5fcec23"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:16:03.432852 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.432832 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed835dd4-48ed-46cf-9180-d616d5fcec23-istio-envoy" pod "ed835dd4-48ed-46cf-9180-d616d5fcec23" (UID: "ed835dd4-48ed-46cf-9180-d616d5fcec23"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:03.432985 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.432971 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed835dd4-48ed-46cf-9180-d616d5fcec23-istio-podinfo" pod "ed835dd4-48ed-46cf-9180-d616d5fcec23" (UID: "ed835dd4-48ed-46cf-9180-d616d5fcec23"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:16:03.511667 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.511636 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f53eb9-d7vlt" Oct 30 20:16:03.511667 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.511659 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f53eb9-d7vlt" event={"ID":"7bbe9c82-995f-4598-a2b6-28b6495f2921","Type":"ContainerDied","Data":"b79be567560354757d3766dc48394958a2f27bbafe5246609cbdaca95b9716d4"} Oct 30 20:16:03.511889 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.511715 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b79be567560354757d3766dc48394958a2f27bbafe5246609cbdaca95b9716d4" Oct 30 20:16:03.513173 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.513147 4292 generic.go:356] "Generic (PLEG): container finished" podID="ed835dd4-48ed-46cf-9180-d616d5fcec23" containerID="db8d85f2e8d793cc7e5b2414cb52537961457cab561346fea480117faf26da92" exitCode=0 Oct 30 20:16:03.513333 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.513186 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49" Oct 30 20:16:03.513333 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.513208 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49" event={"ID":"ed835dd4-48ed-46cf-9180-d616d5fcec23","Type":"ContainerDied","Data":"db8d85f2e8d793cc7e5b2414cb52537961457cab561346fea480117faf26da92"} Oct 30 20:16:03.513333 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.513259 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49" event={"ID":"ed835dd4-48ed-46cf-9180-d616d5fcec23","Type":"ContainerDied","Data":"3521e7579601511a0079c74d02cba5ff38167c4fa87fc0f9728a44be60a75a34"} Oct 30 20:16:03.513333 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.513291 4292 scope.go:122] "RemoveContainer" containerID="db8d85f2e8d793cc7e5b2414cb52537961457cab561346fea480117faf26da92" Oct 30 20:16:03.515234 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.515214 4292 generic.go:356] "Generic (PLEG): container finished" podID="095f29a0-8906-4da7-89ab-95fa4f9dc1a7" containerID="62b7b5d5dc6af96b9cb548456d4ae9c59a0e27ddf2e47a8a86e82708ec218cdd" exitCode=0 Oct 30 20:16:03.515380 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.515256 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd" Oct 30 20:16:03.515380 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.515291 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd" event={"ID":"095f29a0-8906-4da7-89ab-95fa4f9dc1a7","Type":"ContainerDied","Data":"62b7b5d5dc6af96b9cb548456d4ae9c59a0e27ddf2e47a8a86e82708ec218cdd"} Oct 30 20:16:03.515380 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.515327 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd" event={"ID":"095f29a0-8906-4da7-89ab-95fa4f9dc1a7","Type":"ContainerDied","Data":"795b115ea18d749f72842a3b3a4a77eb525d3f6c71679674cba49468d59c5e2e"} Oct 30 20:16:03.526110 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.526081 4292 scope.go:122] "RemoveContainer" containerID="db8d85f2e8d793cc7e5b2414cb52537961457cab561346fea480117faf26da92" Oct 30 20:16:03.526407 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:16:03.526385 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db8d85f2e8d793cc7e5b2414cb52537961457cab561346fea480117faf26da92\": container with ID starting with db8d85f2e8d793cc7e5b2414cb52537961457cab561346fea480117faf26da92 not found: ID does not exist" containerID="db8d85f2e8d793cc7e5b2414cb52537961457cab561346fea480117faf26da92" Oct 30 20:16:03.526500 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.526412 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db8d85f2e8d793cc7e5b2414cb52537961457cab561346fea480117faf26da92"} err="failed to get container status \"db8d85f2e8d793cc7e5b2414cb52537961457cab561346fea480117faf26da92\": rpc error: code = NotFound desc = could not find container \"db8d85f2e8d793cc7e5b2414cb52537961457cab561346fea480117faf26da92\": container with ID starting with db8d85f2e8d793cc7e5b2414cb52537961457cab561346fea480117faf26da92 not found: ID does not exist" Oct 30 20:16:03.526500 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.526430 4292 scope.go:122] "RemoveContainer" containerID="62b7b5d5dc6af96b9cb548456d4ae9c59a0e27ddf2e47a8a86e82708ec218cdd" Oct 30 20:16:03.531218 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.531194 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed835dd4-48ed-46cf-9180-d616d5fcec23-istio-podinfo\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:03.531305 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.531225 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed835dd4-48ed-46cf-9180-d616d5fcec23-istio-token\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:03.531305 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.531243 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62td5\" (UniqueName: \"kubernetes.io/projected/ed835dd4-48ed-46cf-9180-d616d5fcec23-kube-api-access-62td5\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:03.531305 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.531258 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/095f29a0-8906-4da7-89ab-95fa4f9dc1a7-istiod-ca-cert\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:03.531305 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.531273 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed835dd4-48ed-46cf-9180-d616d5fcec23-istio-envoy\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:03.531305 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.531288 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/095f29a0-8906-4da7-89ab-95fa4f9dc1a7-istio-envoy\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:03.531305 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.531300 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/095f29a0-8906-4da7-89ab-95fa4f9dc1a7-istio-token\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:03.531532 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.531315 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/095f29a0-8906-4da7-89ab-95fa4f9dc1a7-istio-podinfo\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:03.531532 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.531344 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtclw\" (UniqueName: \"kubernetes.io/projected/095f29a0-8906-4da7-89ab-95fa4f9dc1a7-kube-api-access-xtclw\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:03.535494 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.535476 4292 scope.go:122] "RemoveContainer" containerID="62b7b5d5dc6af96b9cb548456d4ae9c59a0e27ddf2e47a8a86e82708ec218cdd" Oct 30 20:16:03.535749 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:16:03.535733 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62b7b5d5dc6af96b9cb548456d4ae9c59a0e27ddf2e47a8a86e82708ec218cdd\": container with ID starting with 62b7b5d5dc6af96b9cb548456d4ae9c59a0e27ddf2e47a8a86e82708ec218cdd not found: ID does not exist" containerID="62b7b5d5dc6af96b9cb548456d4ae9c59a0e27ddf2e47a8a86e82708ec218cdd" Oct 30 20:16:03.535813 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.535758 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62b7b5d5dc6af96b9cb548456d4ae9c59a0e27ddf2e47a8a86e82708ec218cdd"} err="failed to get container status \"62b7b5d5dc6af96b9cb548456d4ae9c59a0e27ddf2e47a8a86e82708ec218cdd\": rpc error: code = NotFound desc = could not find container \"62b7b5d5dc6af96b9cb548456d4ae9c59a0e27ddf2e47a8a86e82708ec218cdd\": container with ID starting with 62b7b5d5dc6af96b9cb548456d4ae9c59a0e27ddf2e47a8a86e82708ec218cdd not found: ID does not exist" Oct 30 20:16:03.735594 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.735509 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49"] Oct 30 20:16:03.739518 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.739493 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-tkb49"] Oct 30 20:16:03.870387 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.870339 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd"] Oct 30 20:16:03.873052 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:03.873031 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-wghdd"] Oct 30 20:16:04.582137 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:04.582103 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="095f29a0-8906-4da7-89ab-95fa4f9dc1a7" path="/var/lib/kubelet/pods/095f29a0-8906-4da7-89ab-95fa4f9dc1a7/volumes" Oct 30 20:16:04.582609 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:04.582594 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed835dd4-48ed-46cf-9180-d616d5fcec23" path="/var/lib/kubelet/pods/ed835dd4-48ed-46cf-9180-d616d5fcec23/volumes" Oct 30 20:16:07.523619 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:07.523587 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj"] Oct 30 20:16:07.524144 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:07.523975 4292 cpu_manager.go:414] "containerMap: removing container" podUID="7bbe9c82-995f-4598-a2b6-28b6495f2921" containerName="revoke-keys" Oct 30 20:16:07.524144 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:07.523986 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bbe9c82-995f-4598-a2b6-28b6495f2921" containerName="revoke-keys" Oct 30 20:16:07.524144 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:07.524001 4292 cpu_manager.go:414] "containerMap: removing container" podUID="095f29a0-8906-4da7-89ab-95fa4f9dc1a7" containerName="istio-proxy" Oct 30 20:16:07.524144 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:07.524007 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="095f29a0-8906-4da7-89ab-95fa4f9dc1a7" containerName="istio-proxy" Oct 30 20:16:07.524144 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:07.524029 4292 cpu_manager.go:414] "containerMap: removing container" podUID="ed835dd4-48ed-46cf-9180-d616d5fcec23" containerName="istio-proxy" Oct 30 20:16:07.524144 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:07.524034 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed835dd4-48ed-46cf-9180-d616d5fcec23" containerName="istio-proxy" Oct 30 20:16:07.524144 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:07.524093 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="095f29a0-8906-4da7-89ab-95fa4f9dc1a7" containerName="istio-proxy" Oct 30 20:16:07.524144 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:07.524105 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed835dd4-48ed-46cf-9180-d616d5fcec23" containerName="istio-proxy" Oct 30 20:16:07.524144 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:07.524112 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bbe9c82-995f-4598-a2b6-28b6495f2921" containerName="revoke-keys" Oct 30 20:16:07.528801 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:07.528777 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj" Oct 30 20:16:07.585415 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:07.578069 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f28646cd-7d71-45f8-9a70-407819e9b293-istio-data\") pod \"e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj\" (UID: \"f28646cd-7d71-45f8-9a70-407819e9b293\") " pod="openshift-ingress/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj" Oct 30 20:16:07.585415 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:07.578124 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f28646cd-7d71-45f8-9a70-407819e9b293-istio-podinfo\") pod \"e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj\" (UID: \"f28646cd-7d71-45f8-9a70-407819e9b293\") " pod="openshift-ingress/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj" Oct 30 20:16:07.585415 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:07.578196 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28646cd-7d71-45f8-9a70-407819e9b293-workload-socket\") pod \"e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj\" (UID: \"f28646cd-7d71-45f8-9a70-407819e9b293\") " pod="openshift-ingress/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj" Oct 30 20:16:07.585415 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:07.578226 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcn2s\" (UniqueName: \"kubernetes.io/projected/f28646cd-7d71-45f8-9a70-407819e9b293-kube-api-access-bcn2s\") pod \"e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj\" (UID: \"f28646cd-7d71-45f8-9a70-407819e9b293\") " pod="openshift-ingress/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj" Oct 30 20:16:07.585415 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:07.578255 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28646cd-7d71-45f8-9a70-407819e9b293-credential-socket\") pod \"e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj\" (UID: \"f28646cd-7d71-45f8-9a70-407819e9b293\") " pod="openshift-ingress/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj" Oct 30 20:16:07.585415 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:07.578316 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f28646cd-7d71-45f8-9a70-407819e9b293-workload-certs\") pod \"e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj\" (UID: \"f28646cd-7d71-45f8-9a70-407819e9b293\") " pod="openshift-ingress/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj" Oct 30 20:16:07.585415 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:07.578345 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f28646cd-7d71-45f8-9a70-407819e9b293-istio-envoy\") pod \"e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj\" (UID: \"f28646cd-7d71-45f8-9a70-407819e9b293\") " pod="openshift-ingress/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj" Oct 30 20:16:07.585415 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:07.578491 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f28646cd-7d71-45f8-9a70-407819e9b293-istiod-ca-cert\") pod \"e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj\" (UID: \"f28646cd-7d71-45f8-9a70-407819e9b293\") " pod="openshift-ingress/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj" Oct 30 20:16:07.585415 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:07.578533 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f28646cd-7d71-45f8-9a70-407819e9b293-istio-token\") pod \"e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj\" (UID: \"f28646cd-7d71-45f8-9a70-407819e9b293\") " pod="openshift-ingress/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj" Oct 30 20:16:07.585415 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:07.582982 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-547082-openshift-default-dockercfg-njbbr\"" Oct 30 20:16:07.645625 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:07.645579 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj"] Oct 30 20:16:07.711908 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:07.711868 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f28646cd-7d71-45f8-9a70-407819e9b293-istio-envoy\") pod \"e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj\" (UID: \"f28646cd-7d71-45f8-9a70-407819e9b293\") " pod="openshift-ingress/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj" Oct 30 20:16:07.712100 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:07.712034 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f28646cd-7d71-45f8-9a70-407819e9b293-istiod-ca-cert\") pod \"e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj\" (UID: \"f28646cd-7d71-45f8-9a70-407819e9b293\") " pod="openshift-ingress/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj" Oct 30 20:16:07.712100 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:07.712083 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f28646cd-7d71-45f8-9a70-407819e9b293-istio-token\") pod \"e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj\" (UID: \"f28646cd-7d71-45f8-9a70-407819e9b293\") " pod="openshift-ingress/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj" Oct 30 20:16:07.712224 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:07.712144 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f28646cd-7d71-45f8-9a70-407819e9b293-istio-data\") pod \"e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj\" (UID: \"f28646cd-7d71-45f8-9a70-407819e9b293\") " pod="openshift-ingress/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj" Oct 30 20:16:07.712224 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:07.712168 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f28646cd-7d71-45f8-9a70-407819e9b293-istio-podinfo\") pod \"e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj\" (UID: \"f28646cd-7d71-45f8-9a70-407819e9b293\") " pod="openshift-ingress/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj" Oct 30 20:16:07.712327 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:07.712240 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28646cd-7d71-45f8-9a70-407819e9b293-workload-socket\") pod \"e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj\" (UID: \"f28646cd-7d71-45f8-9a70-407819e9b293\") " pod="openshift-ingress/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj" Oct 30 20:16:07.712327 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:07.712267 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcn2s\" (UniqueName: \"kubernetes.io/projected/f28646cd-7d71-45f8-9a70-407819e9b293-kube-api-access-bcn2s\") pod \"e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj\" (UID: \"f28646cd-7d71-45f8-9a70-407819e9b293\") " pod="openshift-ingress/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj" Oct 30 20:16:07.712450 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:07.712325 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28646cd-7d71-45f8-9a70-407819e9b293-credential-socket\") pod \"e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj\" (UID: \"f28646cd-7d71-45f8-9a70-407819e9b293\") " pod="openshift-ingress/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj" Oct 30 20:16:07.713951 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:07.713923 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f28646cd-7d71-45f8-9a70-407819e9b293-workload-certs\") pod \"e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj\" (UID: \"f28646cd-7d71-45f8-9a70-407819e9b293\") " pod="openshift-ingress/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj" Oct 30 20:16:07.714779 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:07.714755 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f28646cd-7d71-45f8-9a70-407819e9b293-workload-certs\") pod \"e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj\" (UID: \"f28646cd-7d71-45f8-9a70-407819e9b293\") " pod="openshift-ingress/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj" Oct 30 20:16:07.715267 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:07.715247 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f28646cd-7d71-45f8-9a70-407819e9b293-istio-data\") pod \"e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj\" (UID: \"f28646cd-7d71-45f8-9a70-407819e9b293\") " pod="openshift-ingress/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj" Oct 30 20:16:07.715911 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:07.715887 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28646cd-7d71-45f8-9a70-407819e9b293-credential-socket\") pod \"e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj\" (UID: \"f28646cd-7d71-45f8-9a70-407819e9b293\") " pod="openshift-ingress/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj" Oct 30 20:16:07.716037 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:07.715986 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28646cd-7d71-45f8-9a70-407819e9b293-workload-socket\") pod \"e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj\" (UID: \"f28646cd-7d71-45f8-9a70-407819e9b293\") " pod="openshift-ingress/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj" Oct 30 20:16:07.716629 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:07.716604 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f28646cd-7d71-45f8-9a70-407819e9b293-istiod-ca-cert\") pod \"e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj\" (UID: \"f28646cd-7d71-45f8-9a70-407819e9b293\") " pod="openshift-ingress/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj" Oct 30 20:16:07.730655 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:07.719334 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f28646cd-7d71-45f8-9a70-407819e9b293-istio-podinfo\") pod \"e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj\" (UID: \"f28646cd-7d71-45f8-9a70-407819e9b293\") " pod="openshift-ingress/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj" Oct 30 20:16:07.732239 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:07.732215 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f28646cd-7d71-45f8-9a70-407819e9b293-istio-envoy\") pod \"e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj\" (UID: \"f28646cd-7d71-45f8-9a70-407819e9b293\") " pod="openshift-ingress/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj" Oct 30 20:16:07.777992 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:07.777924 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f28646cd-7d71-45f8-9a70-407819e9b293-istio-token\") pod \"e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj\" (UID: \"f28646cd-7d71-45f8-9a70-407819e9b293\") " pod="openshift-ingress/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj" Oct 30 20:16:07.778136 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:07.778104 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcn2s\" (UniqueName: \"kubernetes.io/projected/f28646cd-7d71-45f8-9a70-407819e9b293-kube-api-access-bcn2s\") pod \"e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj\" (UID: \"f28646cd-7d71-45f8-9a70-407819e9b293\") " pod="openshift-ingress/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj" Oct 30 20:16:07.843908 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:07.843866 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj" Oct 30 20:16:08.294288 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:16:08.294256 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf28646cd_7d71_45f8_9a70_407819e9b293.slice/crio-4e5b059fd97399a0d6d85bda73dad99e5b487cb39ab0c1ac5b325bccab10e51e WatchSource:0}: Error finding container 4e5b059fd97399a0d6d85bda73dad99e5b487cb39ab0c1ac5b325bccab10e51e: Status 404 returned error can't find the container with id 4e5b059fd97399a0d6d85bda73dad99e5b487cb39ab0c1ac5b325bccab10e51e Oct 30 20:16:08.296746 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:08.296714 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:16:08.296848 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:08.296781 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:16:08.296848 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:08.296808 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:16:08.297688 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:08.297670 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj"] Oct 30 20:16:08.557428 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:08.556846 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj" event={"ID":"f28646cd-7d71-45f8-9a70-407819e9b293","Type":"ContainerStarted","Data":"814c4b7f8c40ba7416ba75b6addbf79d8f51ba597a7a28de3bdfe43d459c4208"} Oct 30 20:16:08.557428 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:08.556911 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj" event={"ID":"f28646cd-7d71-45f8-9a70-407819e9b293","Type":"ContainerStarted","Data":"4e5b059fd97399a0d6d85bda73dad99e5b487cb39ab0c1ac5b325bccab10e51e"} Oct 30 20:16:08.682470 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:08.682412 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj" podStartSLOduration=1.682395927 podStartE2EDuration="1.682395927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:08.679693083 +0000 UTC m=+3072.198008539" watchObservedRunningTime="2026-10-30 20:16:08.682395927 +0000 UTC m=+3072.200711383" Oct 30 20:16:08.844514 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:08.844442 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj" Oct 30 20:16:09.851366 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:09.851318 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj" Oct 30 20:16:10.570654 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:10.570623 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj" Oct 30 20:16:10.571536 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:10.571513 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj" Oct 30 20:16:13.220578 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:16:13.220543 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:16:25.611141 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:25.611105 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90608da6-729zb"] Oct 30 20:16:25.615234 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:25.615178 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90608da6-729zb" Oct 30 20:16:25.670359 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:25.670316 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:16:25.715805 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:25.715777 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/448bbea1-debf-4e60-be56-98f3f51114d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90608da6-729zb\" (UID: \"448bbea1-debf-4e60-be56-98f3f51114d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90608da6-729zb" Oct 30 20:16:25.715947 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:25.715838 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/448bbea1-debf-4e60-be56-98f3f51114d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90608da6-729zb\" (UID: \"448bbea1-debf-4e60-be56-98f3f51114d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90608da6-729zb" Oct 30 20:16:25.715947 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:25.715926 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qs97\" (UniqueName: \"kubernetes.io/projected/448bbea1-debf-4e60-be56-98f3f51114d8-kube-api-access-6qs97\") pod \"maas-api-revoke-sub-90608da6-729zb\" (UID: \"448bbea1-debf-4e60-be56-98f3f51114d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90608da6-729zb" Oct 30 20:16:25.720723 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:25.720707 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\"" Oct 30 20:16:25.723843 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:25.723823 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90608da6-729zb"] Oct 30 20:16:25.816804 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:25.816779 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qs97\" (UniqueName: \"kubernetes.io/projected/448bbea1-debf-4e60-be56-98f3f51114d8-kube-api-access-6qs97\") pod \"maas-api-revoke-sub-90608da6-729zb\" (UID: \"448bbea1-debf-4e60-be56-98f3f51114d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90608da6-729zb" Oct 30 20:16:25.816967 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:25.816876 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/448bbea1-debf-4e60-be56-98f3f51114d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90608da6-729zb\" (UID: \"448bbea1-debf-4e60-be56-98f3f51114d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90608da6-729zb" Oct 30 20:16:25.816967 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:25.816919 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/448bbea1-debf-4e60-be56-98f3f51114d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90608da6-729zb\" (UID: \"448bbea1-debf-4e60-be56-98f3f51114d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90608da6-729zb" Oct 30 20:16:25.817591 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:25.817564 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/448bbea1-debf-4e60-be56-98f3f51114d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90608da6-729zb\" (UID: \"448bbea1-debf-4e60-be56-98f3f51114d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90608da6-729zb" Oct 30 20:16:25.819481 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:25.819452 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/448bbea1-debf-4e60-be56-98f3f51114d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90608da6-729zb\" (UID: \"448bbea1-debf-4e60-be56-98f3f51114d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90608da6-729zb" Oct 30 20:16:25.880012 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:25.879949 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qs97\" (UniqueName: \"kubernetes.io/projected/448bbea1-debf-4e60-be56-98f3f51114d8-kube-api-access-6qs97\") pod \"maas-api-revoke-sub-90608da6-729zb\" (UID: \"448bbea1-debf-4e60-be56-98f3f51114d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90608da6-729zb" Oct 30 20:16:25.927895 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:25.927870 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90608da6-729zb" Oct 30 20:16:26.302914 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:26.302885 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90608da6-729zb"] Oct 30 20:16:26.303815 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:16:26.303788 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod448bbea1_debf_4e60_be56_98f3f51114d8.slice/crio-9903500f5d1838a564df8d4e382088b0eeafc16b5037b5503d699fd41292f981 WatchSource:0}: Error finding container 9903500f5d1838a564df8d4e382088b0eeafc16b5037b5503d699fd41292f981: Status 404 returned error can't find the container with id 9903500f5d1838a564df8d4e382088b0eeafc16b5037b5503d699fd41292f981 Oct 30 20:16:26.687487 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:26.687447 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90608da6-729zb" event={"ID":"448bbea1-debf-4e60-be56-98f3f51114d8","Type":"ContainerStarted","Data":"8bff9be8e6638bbd1af27154c828615d8a3b0ee15ea9d0bc5369f1aa4fea5687"} Oct 30 20:16:26.687880 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:26.687513 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90608da6-729zb" event={"ID":"448bbea1-debf-4e60-be56-98f3f51114d8","Type":"ContainerStarted","Data":"9903500f5d1838a564df8d4e382088b0eeafc16b5037b5503d699fd41292f981"} Oct 30 20:16:26.808387 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:26.808271 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90608da6-729zb" podStartSLOduration=1.808256089 podStartE2EDuration="1.808256089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:26.805580273 +0000 UTC m=+3090.323895730" watchObservedRunningTime="2026-10-30 20:16:26.808256089 +0000 UTC m=+3090.326571568" Oct 30 20:16:27.057179 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:27.057150 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46c6813c-jmwfh"] Oct 30 20:16:27.060551 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:27.060527 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46c6813c-jmwfh"] Oct 30 20:16:27.695291 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:27.695257 4292 generic.go:356] "Generic (PLEG): container finished" podID="448bbea1-debf-4e60-be56-98f3f51114d8" containerID="8bff9be8e6638bbd1af27154c828615d8a3b0ee15ea9d0bc5369f1aa4fea5687" exitCode=0 Oct 30 20:16:27.695789 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:27.695315 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90608da6-729zb" event={"ID":"448bbea1-debf-4e60-be56-98f3f51114d8","Type":"ContainerDied","Data":"8bff9be8e6638bbd1af27154c828615d8a3b0ee15ea9d0bc5369f1aa4fea5687"} Oct 30 20:16:28.583802 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:28.583773 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f16b42e-55ab-4b16-9450-7a73e5b46a10" path="/var/lib/kubelet/pods/1f16b42e-55ab-4b16-9450-7a73e5b46a10/volumes" Oct 30 20:16:28.870413 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:28.870379 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90608da6-729zb" Oct 30 20:16:28.948419 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:28.948389 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/448bbea1-debf-4e60-be56-98f3f51114d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/448bbea1-debf-4e60-be56-98f3f51114d8-maas-api-xks-ca\") pod \"448bbea1-debf-4e60-be56-98f3f51114d8\" (UID: \"448bbea1-debf-4e60-be56-98f3f51114d8\") " Oct 30 20:16:28.948571 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:28.948469 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/448bbea1-debf-4e60-be56-98f3f51114d8-kube-api-access-6qs97\" (UniqueName: \"kubernetes.io/projected/448bbea1-debf-4e60-be56-98f3f51114d8-kube-api-access-6qs97\") pod \"448bbea1-debf-4e60-be56-98f3f51114d8\" (UID: \"448bbea1-debf-4e60-be56-98f3f51114d8\") " Oct 30 20:16:28.948571 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:28.948550 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/448bbea1-debf-4e60-be56-98f3f51114d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/448bbea1-debf-4e60-be56-98f3f51114d8-maas-api-openshift-ca\") pod \"448bbea1-debf-4e60-be56-98f3f51114d8\" (UID: \"448bbea1-debf-4e60-be56-98f3f51114d8\") " Oct 30 20:16:28.948900 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:28.948877 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/448bbea1-debf-4e60-be56-98f3f51114d8-maas-api-openshift-ca" pod "448bbea1-debf-4e60-be56-98f3f51114d8" (UID: "448bbea1-debf-4e60-be56-98f3f51114d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:16:28.950478 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:28.950452 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/448bbea1-debf-4e60-be56-98f3f51114d8-maas-api-xks-ca" pod "448bbea1-debf-4e60-be56-98f3f51114d8" (UID: "448bbea1-debf-4e60-be56-98f3f51114d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:16:28.950591 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:28.950569 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/448bbea1-debf-4e60-be56-98f3f51114d8-kube-api-access-6qs97" pod "448bbea1-debf-4e60-be56-98f3f51114d8" (UID: "448bbea1-debf-4e60-be56-98f3f51114d8"). InnerVolumeSpecName "kube-api-access-6qs97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:16:29.049697 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:29.049669 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/448bbea1-debf-4e60-be56-98f3f51114d8-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:29.049697 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:29.049696 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/448bbea1-debf-4e60-be56-98f3f51114d8-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:29.049896 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:29.049706 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qs97\" (UniqueName: \"kubernetes.io/projected/448bbea1-debf-4e60-be56-98f3f51114d8-kube-api-access-6qs97\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:29.719090 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:29.719054 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90608da6-729zb" event={"ID":"448bbea1-debf-4e60-be56-98f3f51114d8","Type":"ContainerDied","Data":"9903500f5d1838a564df8d4e382088b0eeafc16b5037b5503d699fd41292f981"} Oct 30 20:16:29.719259 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:29.719118 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9903500f5d1838a564df8d4e382088b0eeafc16b5037b5503d699fd41292f981" Oct 30 20:16:29.719259 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:29.719064 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90608da6-729zb" Oct 30 20:16:32.996321 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:32.992271 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-cr258"] Oct 30 20:16:33.003097 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:33.003067 4292 cpu_manager.go:414] "containerMap: removing container" podUID="448bbea1-debf-4e60-be56-98f3f51114d8" containerName="revoke-keys" Oct 30 20:16:33.003310 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:33.003295 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="448bbea1-debf-4e60-be56-98f3f51114d8" containerName="revoke-keys" Oct 30 20:16:33.003652 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:33.003637 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="448bbea1-debf-4e60-be56-98f3f51114d8" containerName="revoke-keys" Oct 30 20:16:33.012116 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:33.009292 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-cr258" Oct 30 20:16:33.058016 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:33.057983 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-cr258"] Oct 30 20:16:33.115294 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:33.115263 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0f64633-5da3-4c5d-992d-e6254a825c85-istio-token\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-cr258\" (UID: \"a0f64633-5da3-4c5d-992d-e6254a825c85\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-cr258" Oct 30 20:16:33.115535 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:33.115314 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0f64633-5da3-4c5d-992d-e6254a825c85-istiod-ca-cert\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-cr258\" (UID: \"a0f64633-5da3-4c5d-992d-e6254a825c85\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-cr258" Oct 30 20:16:33.115535 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:33.115383 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0f64633-5da3-4c5d-992d-e6254a825c85-workload-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-cr258\" (UID: \"a0f64633-5da3-4c5d-992d-e6254a825c85\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-cr258" Oct 30 20:16:33.115535 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:33.115411 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0f64633-5da3-4c5d-992d-e6254a825c85-istio-envoy\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-cr258\" (UID: \"a0f64633-5da3-4c5d-992d-e6254a825c85\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-cr258" Oct 30 20:16:33.115535 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:33.115443 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0f64633-5da3-4c5d-992d-e6254a825c85-workload-certs\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-cr258\" (UID: \"a0f64633-5da3-4c5d-992d-e6254a825c85\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-cr258" Oct 30 20:16:33.115535 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:33.115487 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0f64633-5da3-4c5d-992d-e6254a825c85-istio-data\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-cr258\" (UID: \"a0f64633-5da3-4c5d-992d-e6254a825c85\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-cr258" Oct 30 20:16:33.115792 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:33.115610 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0f64633-5da3-4c5d-992d-e6254a825c85-credential-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-cr258\" (UID: \"a0f64633-5da3-4c5d-992d-e6254a825c85\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-cr258" Oct 30 20:16:33.115792 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:33.115669 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5dpk\" (UniqueName: \"kubernetes.io/projected/a0f64633-5da3-4c5d-992d-e6254a825c85-kube-api-access-g5dpk\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-cr258\" (UID: \"a0f64633-5da3-4c5d-992d-e6254a825c85\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-cr258" Oct 30 20:16:33.115792 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:33.115765 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0f64633-5da3-4c5d-992d-e6254a825c85-istio-podinfo\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-cr258\" (UID: \"a0f64633-5da3-4c5d-992d-e6254a825c85\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-cr258" Oct 30 20:16:33.216651 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:33.216618 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0f64633-5da3-4c5d-992d-e6254a825c85-istio-token\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-cr258\" (UID: \"a0f64633-5da3-4c5d-992d-e6254a825c85\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-cr258" Oct 30 20:16:33.216651 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:33.216656 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0f64633-5da3-4c5d-992d-e6254a825c85-istiod-ca-cert\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-cr258\" (UID: \"a0f64633-5da3-4c5d-992d-e6254a825c85\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-cr258" Oct 30 20:16:33.216859 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:33.216681 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0f64633-5da3-4c5d-992d-e6254a825c85-workload-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-cr258\" (UID: \"a0f64633-5da3-4c5d-992d-e6254a825c85\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-cr258" Oct 30 20:16:33.216859 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:33.216698 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0f64633-5da3-4c5d-992d-e6254a825c85-istio-envoy\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-cr258\" (UID: \"a0f64633-5da3-4c5d-992d-e6254a825c85\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-cr258" Oct 30 20:16:33.216975 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:33.216867 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0f64633-5da3-4c5d-992d-e6254a825c85-workload-certs\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-cr258\" (UID: \"a0f64633-5da3-4c5d-992d-e6254a825c85\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-cr258" Oct 30 20:16:33.216975 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:33.216949 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0f64633-5da3-4c5d-992d-e6254a825c85-istio-data\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-cr258\" (UID: \"a0f64633-5da3-4c5d-992d-e6254a825c85\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-cr258" Oct 30 20:16:33.217077 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:33.217012 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0f64633-5da3-4c5d-992d-e6254a825c85-credential-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-cr258\" (UID: \"a0f64633-5da3-4c5d-992d-e6254a825c85\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-cr258" Oct 30 20:16:33.217134 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:33.217083 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5dpk\" (UniqueName: \"kubernetes.io/projected/a0f64633-5da3-4c5d-992d-e6254a825c85-kube-api-access-g5dpk\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-cr258\" (UID: \"a0f64633-5da3-4c5d-992d-e6254a825c85\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-cr258" Oct 30 20:16:33.217186 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:33.217130 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0f64633-5da3-4c5d-992d-e6254a825c85-istio-podinfo\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-cr258\" (UID: \"a0f64633-5da3-4c5d-992d-e6254a825c85\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-cr258" Oct 30 20:16:33.217186 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:33.217157 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0f64633-5da3-4c5d-992d-e6254a825c85-workload-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-cr258\" (UID: \"a0f64633-5da3-4c5d-992d-e6254a825c85\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-cr258" Oct 30 20:16:33.217284 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:33.217212 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0f64633-5da3-4c5d-992d-e6254a825c85-workload-certs\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-cr258\" (UID: \"a0f64633-5da3-4c5d-992d-e6254a825c85\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-cr258" Oct 30 20:16:33.217380 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:33.217287 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0f64633-5da3-4c5d-992d-e6254a825c85-istio-data\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-cr258\" (UID: \"a0f64633-5da3-4c5d-992d-e6254a825c85\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-cr258" Oct 30 20:16:33.217562 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:33.217534 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0f64633-5da3-4c5d-992d-e6254a825c85-istiod-ca-cert\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-cr258\" (UID: \"a0f64633-5da3-4c5d-992d-e6254a825c85\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-cr258" Oct 30 20:16:33.217637 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:33.217624 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0f64633-5da3-4c5d-992d-e6254a825c85-credential-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-cr258\" (UID: \"a0f64633-5da3-4c5d-992d-e6254a825c85\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-cr258" Oct 30 20:16:33.218857 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:33.218833 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0f64633-5da3-4c5d-992d-e6254a825c85-istio-envoy\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-cr258\" (UID: \"a0f64633-5da3-4c5d-992d-e6254a825c85\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-cr258" Oct 30 20:16:33.219481 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:33.219464 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0f64633-5da3-4c5d-992d-e6254a825c85-istio-podinfo\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-cr258\" (UID: \"a0f64633-5da3-4c5d-992d-e6254a825c85\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-cr258" Oct 30 20:16:33.279190 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:33.279159 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0f64633-5da3-4c5d-992d-e6254a825c85-istio-token\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-cr258\" (UID: \"a0f64633-5da3-4c5d-992d-e6254a825c85\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-cr258" Oct 30 20:16:33.279320 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:33.279274 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5dpk\" (UniqueName: \"kubernetes.io/projected/a0f64633-5da3-4c5d-992d-e6254a825c85-kube-api-access-g5dpk\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-cr258\" (UID: \"a0f64633-5da3-4c5d-992d-e6254a825c85\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-cr258" Oct 30 20:16:33.328417 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:33.328385 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-cr258" Oct 30 20:16:33.763582 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:33.763551 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-cr258"] Oct 30 20:16:33.764715 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:16:33.764665 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0f64633_5da3_4c5d_992d_e6254a825c85.slice/crio-0ec3fdd349e5c90098ab38af4d7a4890c4f4682d3ff6bded72c6c6dba7fde3fc WatchSource:0}: Error finding container 0ec3fdd349e5c90098ab38af4d7a4890c4f4682d3ff6bded72c6c6dba7fde3fc: Status 404 returned error can't find the container with id 0ec3fdd349e5c90098ab38af4d7a4890c4f4682d3ff6bded72c6c6dba7fde3fc Oct 30 20:16:33.768019 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:33.767977 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:16:33.768126 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:33.768063 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:16:33.768126 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:33.768111 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:16:34.648251 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:34.648219 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f53eb9-d7vlt"] Oct 30 20:16:34.662376 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:34.651978 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f53eb9-d7vlt"] Oct 30 20:16:34.776794 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:34.776752 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-cr258" event={"ID":"a0f64633-5da3-4c5d-992d-e6254a825c85","Type":"ContainerStarted","Data":"905bd0a8aed4d3fef2a848e2fd395486f034de41443b7217b230c84a5dec3d02"} Oct 30 20:16:34.777042 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:34.777022 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-cr258" event={"ID":"a0f64633-5da3-4c5d-992d-e6254a825c85","Type":"ContainerStarted","Data":"0ec3fdd349e5c90098ab38af4d7a4890c4f4682d3ff6bded72c6c6dba7fde3fc"} Oct 30 20:16:34.915431 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:34.911994 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-cr258" podStartSLOduration=2.911974614 podStartE2EDuration="2.911974614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:34.90169872 +0000 UTC m=+3098.420014179" watchObservedRunningTime="2026-10-30 20:16:34.911974614 +0000 UTC m=+3098.430290072" Oct 30 20:16:35.328552 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:35.328516 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-cr258" Oct 30 20:16:35.330134 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:35.330102 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-cr258 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Oct 30 20:16:35.330316 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:35.330171 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-cr258" podUID="a0f64633-5da3-4c5d-992d-e6254a825c85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Oct 30 20:16:36.329042 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:36.329008 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-cr258 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Oct 30 20:16:36.329480 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:36.329072 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-cr258" podUID="a0f64633-5da3-4c5d-992d-e6254a825c85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Oct 30 20:16:36.592046 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:36.591975 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bbe9c82-995f-4598-a2b6-28b6495f2921" path="/var/lib/kubelet/pods/7bbe9c82-995f-4598-a2b6-28b6495f2921/volumes" Oct 30 20:16:37.329953 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:37.329919 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-cr258 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Oct 30 20:16:37.330635 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:37.330600 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-cr258" podUID="a0f64633-5da3-4c5d-992d-e6254a825c85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Oct 30 20:16:38.329165 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:38.329126 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-cr258 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Oct 30 20:16:38.329468 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:38.329183 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-cr258" podUID="a0f64633-5da3-4c5d-992d-e6254a825c85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Oct 30 20:16:39.335090 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:39.335051 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-cr258 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Oct 30 20:16:39.335604 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:39.335572 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-cr258" podUID="a0f64633-5da3-4c5d-992d-e6254a825c85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Oct 30 20:16:40.329868 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:40.329784 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-cr258 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Oct 30 20:16:40.329868 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:40.329853 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-cr258" podUID="a0f64633-5da3-4c5d-992d-e6254a825c85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Oct 30 20:16:41.335026 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:41.334996 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-cr258" Oct 30 20:16:41.335562 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:41.335542 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-cr258" Oct 30 20:16:41.336390 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:41.336342 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-cr258" Oct 30 20:16:41.632669 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:41.632588 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp"] Oct 30 20:16:41.632835 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:41.632804 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp" podUID="9543233d-9115-4459-87b7-dea47aa8d1ca" containerName="istio-proxy" containerID="cri-o://7853d32bb00ffae07b1f5a18938660074982e78890b8f0d7e2c9745c84cef977" gracePeriod=30 Oct 30 20:16:43.222640 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:16:43.222595 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:16:43.842438 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:43.842403 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh"] Oct 30 20:16:43.847412 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:43.847390 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh" Oct 30 20:16:43.895638 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:43.895597 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn"] Oct 30 20:16:43.900239 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:43.900212 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn" Oct 30 20:16:43.901697 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:43.901662 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4f584f0-f6b6-474d-ba63-967f27c5a8ff-istio-podinfo\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn\" (UID: \"a4f584f0-f6b6-474d-ba63-967f27c5a8ff\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn" Oct 30 20:16:43.901828 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:43.901717 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec42d060-8ef1-46d2-9bb8-4ebb4224389b-istio-data\") pod \"e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh\" (UID: \"ec42d060-8ef1-46d2-9bb8-4ebb4224389b\") " pod="openshift-ingress/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh" Oct 30 20:16:43.901828 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:43.901745 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49m8k\" (UniqueName: \"kubernetes.io/projected/a4f584f0-f6b6-474d-ba63-967f27c5a8ff-kube-api-access-49m8k\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn\" (UID: \"a4f584f0-f6b6-474d-ba63-967f27c5a8ff\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn" Oct 30 20:16:43.901955 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:43.901863 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec42d060-8ef1-46d2-9bb8-4ebb4224389b-istio-envoy\") pod \"e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh\" (UID: \"ec42d060-8ef1-46d2-9bb8-4ebb4224389b\") " pod="openshift-ingress/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh" Oct 30 20:16:43.901955 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:43.901902 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec42d060-8ef1-46d2-9bb8-4ebb4224389b-workload-socket\") pod \"e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh\" (UID: \"ec42d060-8ef1-46d2-9bb8-4ebb4224389b\") " pod="openshift-ingress/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh" Oct 30 20:16:43.901955 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:43.901926 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f584f0-f6b6-474d-ba63-967f27c5a8ff-workload-socket\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn\" (UID: \"a4f584f0-f6b6-474d-ba63-967f27c5a8ff\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn" Oct 30 20:16:43.901955 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:43.901951 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec42d060-8ef1-46d2-9bb8-4ebb4224389b-istio-token\") pod \"e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh\" (UID: \"ec42d060-8ef1-46d2-9bb8-4ebb4224389b\") " pod="openshift-ingress/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh" Oct 30 20:16:43.902172 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:43.901969 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvhx8\" (UniqueName: \"kubernetes.io/projected/ec42d060-8ef1-46d2-9bb8-4ebb4224389b-kube-api-access-gvhx8\") pod \"e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh\" (UID: \"ec42d060-8ef1-46d2-9bb8-4ebb4224389b\") " pod="openshift-ingress/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh" Oct 30 20:16:43.902172 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:43.901984 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f584f0-f6b6-474d-ba63-967f27c5a8ff-credential-socket\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn\" (UID: \"a4f584f0-f6b6-474d-ba63-967f27c5a8ff\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn" Oct 30 20:16:43.902172 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:43.902001 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4f584f0-f6b6-474d-ba63-967f27c5a8ff-istio-token\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn\" (UID: \"a4f584f0-f6b6-474d-ba63-967f27c5a8ff\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn" Oct 30 20:16:43.902172 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:43.902028 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4f584f0-f6b6-474d-ba63-967f27c5a8ff-workload-certs\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn\" (UID: \"a4f584f0-f6b6-474d-ba63-967f27c5a8ff\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn" Oct 30 20:16:43.902172 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:43.902054 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec42d060-8ef1-46d2-9bb8-4ebb4224389b-workload-certs\") pod \"e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh\" (UID: \"ec42d060-8ef1-46d2-9bb8-4ebb4224389b\") " pod="openshift-ingress/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh" Oct 30 20:16:43.902172 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:43.902070 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec42d060-8ef1-46d2-9bb8-4ebb4224389b-istiod-ca-cert\") pod \"e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh\" (UID: \"ec42d060-8ef1-46d2-9bb8-4ebb4224389b\") " pod="openshift-ingress/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh" Oct 30 20:16:43.902172 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:43.902086 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4f584f0-f6b6-474d-ba63-967f27c5a8ff-istio-data\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn\" (UID: \"a4f584f0-f6b6-474d-ba63-967f27c5a8ff\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn" Oct 30 20:16:43.902172 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:43.902111 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec42d060-8ef1-46d2-9bb8-4ebb4224389b-istio-podinfo\") pod \"e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh\" (UID: \"ec42d060-8ef1-46d2-9bb8-4ebb4224389b\") " pod="openshift-ingress/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh" Oct 30 20:16:43.902172 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:43.902125 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4f584f0-f6b6-474d-ba63-967f27c5a8ff-istio-envoy\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn\" (UID: \"a4f584f0-f6b6-474d-ba63-967f27c5a8ff\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn" Oct 30 20:16:43.902172 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:43.902160 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4f584f0-f6b6-474d-ba63-967f27c5a8ff-istiod-ca-cert\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn\" (UID: \"a4f584f0-f6b6-474d-ba63-967f27c5a8ff\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn" Oct 30 20:16:43.902750 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:43.902189 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec42d060-8ef1-46d2-9bb8-4ebb4224389b-credential-socket\") pod \"e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh\" (UID: \"ec42d060-8ef1-46d2-9bb8-4ebb4224389b\") " pod="openshift-ingress/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh" Oct 30 20:16:43.946116 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:43.946083 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh"] Oct 30 20:16:43.983412 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:43.983382 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-056a27-openshift-default-dockercfg-wxv97\"" Oct 30 20:16:44.017530 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:44.017492 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4f584f0-f6b6-474d-ba63-967f27c5a8ff-istio-podinfo\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn\" (UID: \"a4f584f0-f6b6-474d-ba63-967f27c5a8ff\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn" Oct 30 20:16:44.017792 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:44.017770 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec42d060-8ef1-46d2-9bb8-4ebb4224389b-istio-data\") pod \"e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh\" (UID: \"ec42d060-8ef1-46d2-9bb8-4ebb4224389b\") " pod="openshift-ingress/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh" Oct 30 20:16:44.017923 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:44.017906 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49m8k\" (UniqueName: \"kubernetes.io/projected/a4f584f0-f6b6-474d-ba63-967f27c5a8ff-kube-api-access-49m8k\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn\" (UID: \"a4f584f0-f6b6-474d-ba63-967f27c5a8ff\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn" Oct 30 20:16:44.018067 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:44.018049 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec42d060-8ef1-46d2-9bb8-4ebb4224389b-istio-envoy\") pod \"e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh\" (UID: \"ec42d060-8ef1-46d2-9bb8-4ebb4224389b\") " pod="openshift-ingress/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh" Oct 30 20:16:44.018185 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:44.018169 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec42d060-8ef1-46d2-9bb8-4ebb4224389b-workload-socket\") pod \"e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh\" (UID: \"ec42d060-8ef1-46d2-9bb8-4ebb4224389b\") " pod="openshift-ingress/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh" Oct 30 20:16:44.018310 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:44.018295 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f584f0-f6b6-474d-ba63-967f27c5a8ff-workload-socket\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn\" (UID: \"a4f584f0-f6b6-474d-ba63-967f27c5a8ff\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn" Oct 30 20:16:44.018481 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:44.018458 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec42d060-8ef1-46d2-9bb8-4ebb4224389b-istio-token\") pod \"e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh\" (UID: \"ec42d060-8ef1-46d2-9bb8-4ebb4224389b\") " pod="openshift-ingress/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh" Oct 30 20:16:44.018609 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:44.018594 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvhx8\" (UniqueName: \"kubernetes.io/projected/ec42d060-8ef1-46d2-9bb8-4ebb4224389b-kube-api-access-gvhx8\") pod \"e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh\" (UID: \"ec42d060-8ef1-46d2-9bb8-4ebb4224389b\") " pod="openshift-ingress/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh" Oct 30 20:16:44.018715 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:44.018701 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f584f0-f6b6-474d-ba63-967f27c5a8ff-credential-socket\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn\" (UID: \"a4f584f0-f6b6-474d-ba63-967f27c5a8ff\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn" Oct 30 20:16:44.018817 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:44.018798 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4f584f0-f6b6-474d-ba63-967f27c5a8ff-istio-token\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn\" (UID: \"a4f584f0-f6b6-474d-ba63-967f27c5a8ff\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn" Oct 30 20:16:44.018933 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:44.018920 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4f584f0-f6b6-474d-ba63-967f27c5a8ff-workload-certs\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn\" (UID: \"a4f584f0-f6b6-474d-ba63-967f27c5a8ff\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn" Oct 30 20:16:44.019068 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:44.019054 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec42d060-8ef1-46d2-9bb8-4ebb4224389b-workload-certs\") pod \"e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh\" (UID: \"ec42d060-8ef1-46d2-9bb8-4ebb4224389b\") " pod="openshift-ingress/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh" Oct 30 20:16:44.019176 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:44.019163 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec42d060-8ef1-46d2-9bb8-4ebb4224389b-istiod-ca-cert\") pod \"e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh\" (UID: \"ec42d060-8ef1-46d2-9bb8-4ebb4224389b\") " pod="openshift-ingress/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh" Oct 30 20:16:44.019272 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:44.019260 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4f584f0-f6b6-474d-ba63-967f27c5a8ff-istio-data\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn\" (UID: \"a4f584f0-f6b6-474d-ba63-967f27c5a8ff\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn" Oct 30 20:16:44.019413 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:44.019396 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec42d060-8ef1-46d2-9bb8-4ebb4224389b-istio-podinfo\") pod \"e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh\" (UID: \"ec42d060-8ef1-46d2-9bb8-4ebb4224389b\") " pod="openshift-ingress/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh" Oct 30 20:16:44.019522 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:44.019508 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4f584f0-f6b6-474d-ba63-967f27c5a8ff-istio-envoy\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn\" (UID: \"a4f584f0-f6b6-474d-ba63-967f27c5a8ff\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn" Oct 30 20:16:44.019646 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:44.019633 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4f584f0-f6b6-474d-ba63-967f27c5a8ff-istiod-ca-cert\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn\" (UID: \"a4f584f0-f6b6-474d-ba63-967f27c5a8ff\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn" Oct 30 20:16:44.019757 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:44.019745 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec42d060-8ef1-46d2-9bb8-4ebb4224389b-credential-socket\") pod \"e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh\" (UID: \"ec42d060-8ef1-46d2-9bb8-4ebb4224389b\") " pod="openshift-ingress/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh" Oct 30 20:16:44.020310 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:44.020285 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec42d060-8ef1-46d2-9bb8-4ebb4224389b-credential-socket\") pod \"e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh\" (UID: \"ec42d060-8ef1-46d2-9bb8-4ebb4224389b\") " pod="openshift-ingress/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh" Oct 30 20:16:44.021161 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:44.021135 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f584f0-f6b6-474d-ba63-967f27c5a8ff-credential-socket\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn\" (UID: \"a4f584f0-f6b6-474d-ba63-967f27c5a8ff\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn" Oct 30 20:16:44.022175 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:44.022152 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec42d060-8ef1-46d2-9bb8-4ebb4224389b-istio-data\") pod \"e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh\" (UID: \"ec42d060-8ef1-46d2-9bb8-4ebb4224389b\") " pod="openshift-ingress/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh" Oct 30 20:16:44.022322 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:44.022192 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec42d060-8ef1-46d2-9bb8-4ebb4224389b-workload-socket\") pod \"e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh\" (UID: \"ec42d060-8ef1-46d2-9bb8-4ebb4224389b\") " pod="openshift-ingress/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh" Oct 30 20:16:44.022724 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:44.022704 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f584f0-f6b6-474d-ba63-967f27c5a8ff-workload-socket\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn\" (UID: \"a4f584f0-f6b6-474d-ba63-967f27c5a8ff\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn" Oct 30 20:16:44.024417 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:44.023187 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4f584f0-f6b6-474d-ba63-967f27c5a8ff-workload-certs\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn\" (UID: \"a4f584f0-f6b6-474d-ba63-967f27c5a8ff\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn" Oct 30 20:16:44.024553 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:44.023427 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4f584f0-f6b6-474d-ba63-967f27c5a8ff-istio-data\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn\" (UID: \"a4f584f0-f6b6-474d-ba63-967f27c5a8ff\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn" Oct 30 20:16:44.024672 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:44.023669 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec42d060-8ef1-46d2-9bb8-4ebb4224389b-workload-certs\") pod \"e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh\" (UID: \"ec42d060-8ef1-46d2-9bb8-4ebb4224389b\") " pod="openshift-ingress/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh" Oct 30 20:16:44.024813 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:44.024335 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4f584f0-f6b6-474d-ba63-967f27c5a8ff-istiod-ca-cert\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn\" (UID: \"a4f584f0-f6b6-474d-ba63-967f27c5a8ff\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn" Oct 30 20:16:44.024936 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:44.024482 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn"] Oct 30 20:16:44.025753 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:44.025731 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec42d060-8ef1-46d2-9bb8-4ebb4224389b-istiod-ca-cert\") pod \"e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh\" (UID: \"ec42d060-8ef1-46d2-9bb8-4ebb4224389b\") " pod="openshift-ingress/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh" Oct 30 20:16:44.028460 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:44.028408 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec42d060-8ef1-46d2-9bb8-4ebb4224389b-istio-podinfo\") pod \"e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh\" (UID: \"ec42d060-8ef1-46d2-9bb8-4ebb4224389b\") " pod="openshift-ingress/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh" Oct 30 20:16:44.029078 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:44.029057 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4f584f0-f6b6-474d-ba63-967f27c5a8ff-istio-envoy\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn\" (UID: \"a4f584f0-f6b6-474d-ba63-967f27c5a8ff\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn" Oct 30 20:16:44.032107 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:44.031914 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4f584f0-f6b6-474d-ba63-967f27c5a8ff-istio-podinfo\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn\" (UID: \"a4f584f0-f6b6-474d-ba63-967f27c5a8ff\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn" Oct 30 20:16:44.032107 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:44.032075 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec42d060-8ef1-46d2-9bb8-4ebb4224389b-istio-envoy\") pod \"e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh\" (UID: \"ec42d060-8ef1-46d2-9bb8-4ebb4224389b\") " pod="openshift-ingress/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh" Oct 30 20:16:44.107087 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:44.106979 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49m8k\" (UniqueName: \"kubernetes.io/projected/a4f584f0-f6b6-474d-ba63-967f27c5a8ff-kube-api-access-49m8k\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn\" (UID: \"a4f584f0-f6b6-474d-ba63-967f27c5a8ff\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn" Oct 30 20:16:44.107836 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:44.107779 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4f584f0-f6b6-474d-ba63-967f27c5a8ff-istio-token\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn\" (UID: \"a4f584f0-f6b6-474d-ba63-967f27c5a8ff\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn" Oct 30 20:16:44.110772 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:44.110721 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec42d060-8ef1-46d2-9bb8-4ebb4224389b-istio-token\") pod \"e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh\" (UID: \"ec42d060-8ef1-46d2-9bb8-4ebb4224389b\") " pod="openshift-ingress/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh" Oct 30 20:16:44.126506 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:44.126401 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvhx8\" (UniqueName: \"kubernetes.io/projected/ec42d060-8ef1-46d2-9bb8-4ebb4224389b-kube-api-access-gvhx8\") pod \"e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh\" (UID: \"ec42d060-8ef1-46d2-9bb8-4ebb4224389b\") " pod="openshift-ingress/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh" Oct 30 20:16:44.161534 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:44.161500 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh" Oct 30 20:16:44.215730 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:44.215488 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn" Oct 30 20:16:44.640129 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:16:44.640094 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec42d060_8ef1_46d2_9bb8_4ebb4224389b.slice/crio-b1b7406f5cdd55f97301cdf56dff12b7f6ad72029c4776fdea8a6187226452f7 WatchSource:0}: Error finding container b1b7406f5cdd55f97301cdf56dff12b7f6ad72029c4776fdea8a6187226452f7: Status 404 returned error can't find the container with id b1b7406f5cdd55f97301cdf56dff12b7f6ad72029c4776fdea8a6187226452f7 Oct 30 20:16:44.643537 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:44.642771 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:16:44.643537 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:44.642837 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:16:44.643537 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:44.642868 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:16:44.673419 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:44.653513 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh"] Oct 30 20:16:44.674787 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:44.674758 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn"] Oct 30 20:16:44.690398 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:44.690338 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:16:44.690526 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:44.690451 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:16:44.690526 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:44.690493 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:16:44.970059 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:44.969974 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn" event={"ID":"a4f584f0-f6b6-474d-ba63-967f27c5a8ff","Type":"ContainerStarted","Data":"6c2d4f9573a7c906f3da701cca3bf5245581487db8f35ad991b1b28cf5e2dd6f"} Oct 30 20:16:44.970460 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:44.970433 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn" event={"ID":"a4f584f0-f6b6-474d-ba63-967f27c5a8ff","Type":"ContainerStarted","Data":"ece8675f4402c683bedb92e721752af77e57b6acc837be99efe38185e259eb6d"} Oct 30 20:16:44.973977 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:44.973949 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh" event={"ID":"ec42d060-8ef1-46d2-9bb8-4ebb4224389b","Type":"ContainerStarted","Data":"fa7e62302e8ab4f89ed8e1c71a650a09ad421e825e4239ec7e711a8dbe778d92"} Oct 30 20:16:44.974147 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:44.974131 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh" event={"ID":"ec42d060-8ef1-46d2-9bb8-4ebb4224389b","Type":"ContainerStarted","Data":"b1b7406f5cdd55f97301cdf56dff12b7f6ad72029c4776fdea8a6187226452f7"} Oct 30 20:16:45.099536 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:45.099472 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn" podStartSLOduration=2.099450679 podStartE2EDuration="2.099450679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:45.095393804 +0000 UTC m=+3108.613709287" watchObservedRunningTime="2026-10-30 20:16:45.099450679 +0000 UTC m=+3108.617766139" Oct 30 20:16:45.161639 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:45.161601 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh" Oct 30 20:16:45.163108 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:45.163079 4292 patch_prober.go:29] interesting pod/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Oct 30 20:16:45.163248 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:45.163148 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh" podUID="ec42d060-8ef1-46d2-9bb8-4ebb4224389b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Oct 30 20:16:45.216580 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:45.216543 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn" Oct 30 20:16:45.218555 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:45.218522 4292 patch_prober.go:29] interesting pod/e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Oct 30 20:16:45.218694 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:45.218602 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn" podUID="a4f584f0-f6b6-474d-ba63-967f27c5a8ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Oct 30 20:16:45.221688 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:45.221578 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh" podStartSLOduration=2.221562299 podStartE2EDuration="2.221562299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:45.219922415 +0000 UTC m=+3108.738237872" watchObservedRunningTime="2026-10-30 20:16:45.221562299 +0000 UTC m=+3108.739877757" Oct 30 20:16:46.209540 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:46.209509 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh" Oct 30 20:16:46.216853 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:46.216824 4292 patch_prober.go:29] interesting pod/e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Oct 30 20:16:46.216994 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:46.216878 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn" podUID="a4f584f0-f6b6-474d-ba63-967f27c5a8ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Oct 30 20:16:46.944296 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:46.944267 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp" Oct 30 20:16:46.989665 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:46.989626 4292 generic.go:356] "Generic (PLEG): container finished" podID="9543233d-9115-4459-87b7-dea47aa8d1ca" containerID="7853d32bb00ffae07b1f5a18938660074982e78890b8f0d7e2c9745c84cef977" exitCode=0 Oct 30 20:16:46.989827 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:46.989684 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp" Oct 30 20:16:46.989827 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:46.989709 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp" event={"ID":"9543233d-9115-4459-87b7-dea47aa8d1ca","Type":"ContainerDied","Data":"7853d32bb00ffae07b1f5a18938660074982e78890b8f0d7e2c9745c84cef977"} Oct 30 20:16:46.989827 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:46.989770 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp" event={"ID":"9543233d-9115-4459-87b7-dea47aa8d1ca","Type":"ContainerDied","Data":"dd383518e772719114c20f50c5628fb8df910035bcd2b15f71afd209e0360706"} Oct 30 20:16:46.989827 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:46.989794 4292 scope.go:122] "RemoveContainer" containerID="7853d32bb00ffae07b1f5a18938660074982e78890b8f0d7e2c9745c84cef977" Oct 30 20:16:46.990628 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:46.990594 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh" Oct 30 20:16:46.991195 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:46.991179 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh" Oct 30 20:16:46.995760 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:46.995730 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9543233d-9115-4459-87b7-dea47aa8d1ca-istio-token\" (UniqueName: \"kubernetes.io/projected/9543233d-9115-4459-87b7-dea47aa8d1ca-istio-token\") pod \"9543233d-9115-4459-87b7-dea47aa8d1ca\" (UID: \"9543233d-9115-4459-87b7-dea47aa8d1ca\") " Oct 30 20:16:46.995862 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:46.995825 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9543233d-9115-4459-87b7-dea47aa8d1ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9543233d-9115-4459-87b7-dea47aa8d1ca-credential-socket\") pod \"9543233d-9115-4459-87b7-dea47aa8d1ca\" (UID: \"9543233d-9115-4459-87b7-dea47aa8d1ca\") " Oct 30 20:16:46.995921 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:46.995882 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9543233d-9115-4459-87b7-dea47aa8d1ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9543233d-9115-4459-87b7-dea47aa8d1ca-workload-socket\") pod \"9543233d-9115-4459-87b7-dea47aa8d1ca\" (UID: \"9543233d-9115-4459-87b7-dea47aa8d1ca\") " Oct 30 20:16:46.995976 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:46.995918 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9543233d-9115-4459-87b7-dea47aa8d1ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9543233d-9115-4459-87b7-dea47aa8d1ca-istio-envoy\") pod \"9543233d-9115-4459-87b7-dea47aa8d1ca\" (UID: \"9543233d-9115-4459-87b7-dea47aa8d1ca\") " Oct 30 20:16:46.995976 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:46.995946 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9543233d-9115-4459-87b7-dea47aa8d1ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9543233d-9115-4459-87b7-dea47aa8d1ca-istio-data\") pod \"9543233d-9115-4459-87b7-dea47aa8d1ca\" (UID: \"9543233d-9115-4459-87b7-dea47aa8d1ca\") " Oct 30 20:16:46.996078 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:46.995980 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9543233d-9115-4459-87b7-dea47aa8d1ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9543233d-9115-4459-87b7-dea47aa8d1ca-istio-podinfo\") pod \"9543233d-9115-4459-87b7-dea47aa8d1ca\" (UID: \"9543233d-9115-4459-87b7-dea47aa8d1ca\") " Oct 30 20:16:46.996078 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:46.996011 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9543233d-9115-4459-87b7-dea47aa8d1ca-kube-api-access-q5cq9\" (UniqueName: \"kubernetes.io/projected/9543233d-9115-4459-87b7-dea47aa8d1ca-kube-api-access-q5cq9\") pod \"9543233d-9115-4459-87b7-dea47aa8d1ca\" (UID: \"9543233d-9115-4459-87b7-dea47aa8d1ca\") " Oct 30 20:16:46.996078 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:46.996038 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9543233d-9115-4459-87b7-dea47aa8d1ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9543233d-9115-4459-87b7-dea47aa8d1ca-istiod-ca-cert\") pod \"9543233d-9115-4459-87b7-dea47aa8d1ca\" (UID: \"9543233d-9115-4459-87b7-dea47aa8d1ca\") " Oct 30 20:16:46.996221 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:46.996112 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9543233d-9115-4459-87b7-dea47aa8d1ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9543233d-9115-4459-87b7-dea47aa8d1ca-workload-certs\") pod \"9543233d-9115-4459-87b7-dea47aa8d1ca\" (UID: \"9543233d-9115-4459-87b7-dea47aa8d1ca\") " Oct 30 20:16:46.996221 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:46.996135 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9543233d-9115-4459-87b7-dea47aa8d1ca-credential-socket" pod "9543233d-9115-4459-87b7-dea47aa8d1ca" (UID: "9543233d-9115-4459-87b7-dea47aa8d1ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:46.996221 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:46.996200 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9543233d-9115-4459-87b7-dea47aa8d1ca-istio-data" pod "9543233d-9115-4459-87b7-dea47aa8d1ca" (UID: "9543233d-9115-4459-87b7-dea47aa8d1ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:46.996470 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:46.996447 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9543233d-9115-4459-87b7-dea47aa8d1ca-workload-certs" pod "9543233d-9115-4459-87b7-dea47aa8d1ca" (UID: "9543233d-9115-4459-87b7-dea47aa8d1ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:46.996531 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:46.996509 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9543233d-9115-4459-87b7-dea47aa8d1ca-istiod-ca-cert" pod "9543233d-9115-4459-87b7-dea47aa8d1ca" (UID: "9543233d-9115-4459-87b7-dea47aa8d1ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:16:46.996818 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:46.996797 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9543233d-9115-4459-87b7-dea47aa8d1ca-workload-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:46.996876 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:46.996827 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9543233d-9115-4459-87b7-dea47aa8d1ca-credential-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:46.996876 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:46.996842 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9543233d-9115-4459-87b7-dea47aa8d1ca-istio-data\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:46.996876 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:46.996858 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9543233d-9115-4459-87b7-dea47aa8d1ca-istiod-ca-cert\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:46.997255 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:46.997236 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9543233d-9115-4459-87b7-dea47aa8d1ca-workload-socket" pod "9543233d-9115-4459-87b7-dea47aa8d1ca" (UID: "9543233d-9115-4459-87b7-dea47aa8d1ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:46.998788 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:46.998764 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9543233d-9115-4459-87b7-dea47aa8d1ca-istio-envoy" pod "9543233d-9115-4459-87b7-dea47aa8d1ca" (UID: "9543233d-9115-4459-87b7-dea47aa8d1ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:46.999641 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:46.999618 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9543233d-9115-4459-87b7-dea47aa8d1ca-istio-podinfo" pod "9543233d-9115-4459-87b7-dea47aa8d1ca" (UID: "9543233d-9115-4459-87b7-dea47aa8d1ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:16:46.999732 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:46.999640 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9543233d-9115-4459-87b7-dea47aa8d1ca-istio-token" pod "9543233d-9115-4459-87b7-dea47aa8d1ca" (UID: "9543233d-9115-4459-87b7-dea47aa8d1ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:16:47.001206 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:47.001178 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9543233d-9115-4459-87b7-dea47aa8d1ca-kube-api-access-q5cq9" pod "9543233d-9115-4459-87b7-dea47aa8d1ca" (UID: "9543233d-9115-4459-87b7-dea47aa8d1ca"). InnerVolumeSpecName "kube-api-access-q5cq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:16:47.003064 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:47.003043 4292 scope.go:122] "RemoveContainer" containerID="8eaea5d77e405b49dc56c4751bd2ca3f9eb40b796c0e683291ea3a8780f150d5" Oct 30 20:16:47.021236 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:47.021216 4292 scope.go:122] "RemoveContainer" containerID="7853d32bb00ffae07b1f5a18938660074982e78890b8f0d7e2c9745c84cef977" Oct 30 20:16:47.021580 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:16:47.021561 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7853d32bb00ffae07b1f5a18938660074982e78890b8f0d7e2c9745c84cef977\": container with ID starting with 7853d32bb00ffae07b1f5a18938660074982e78890b8f0d7e2c9745c84cef977 not found: ID does not exist" containerID="7853d32bb00ffae07b1f5a18938660074982e78890b8f0d7e2c9745c84cef977" Oct 30 20:16:47.021654 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:47.021588 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7853d32bb00ffae07b1f5a18938660074982e78890b8f0d7e2c9745c84cef977"} err="failed to get container status \"7853d32bb00ffae07b1f5a18938660074982e78890b8f0d7e2c9745c84cef977\": rpc error: code = NotFound desc = could not find container \"7853d32bb00ffae07b1f5a18938660074982e78890b8f0d7e2c9745c84cef977\": container with ID starting with 7853d32bb00ffae07b1f5a18938660074982e78890b8f0d7e2c9745c84cef977 not found: ID does not exist" Oct 30 20:16:47.021654 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:47.021610 4292 scope.go:122] "RemoveContainer" containerID="8eaea5d77e405b49dc56c4751bd2ca3f9eb40b796c0e683291ea3a8780f150d5" Oct 30 20:16:47.021896 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:16:47.021874 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eaea5d77e405b49dc56c4751bd2ca3f9eb40b796c0e683291ea3a8780f150d5\": container with ID starting with 8eaea5d77e405b49dc56c4751bd2ca3f9eb40b796c0e683291ea3a8780f150d5 not found: ID does not exist" containerID="8eaea5d77e405b49dc56c4751bd2ca3f9eb40b796c0e683291ea3a8780f150d5" Oct 30 20:16:47.021968 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:47.021905 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eaea5d77e405b49dc56c4751bd2ca3f9eb40b796c0e683291ea3a8780f150d5"} err="failed to get container status \"8eaea5d77e405b49dc56c4751bd2ca3f9eb40b796c0e683291ea3a8780f150d5\": rpc error: code = NotFound desc = could not find container \"8eaea5d77e405b49dc56c4751bd2ca3f9eb40b796c0e683291ea3a8780f150d5\": container with ID starting with 8eaea5d77e405b49dc56c4751bd2ca3f9eb40b796c0e683291ea3a8780f150d5 not found: ID does not exist" Oct 30 20:16:47.098381 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:47.098335 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9543233d-9115-4459-87b7-dea47aa8d1ca-workload-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:47.098381 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:47.098379 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9543233d-9115-4459-87b7-dea47aa8d1ca-istio-envoy\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:47.098604 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:47.098392 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9543233d-9115-4459-87b7-dea47aa8d1ca-istio-podinfo\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:47.098604 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:47.098405 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5cq9\" (UniqueName: \"kubernetes.io/projected/9543233d-9115-4459-87b7-dea47aa8d1ca-kube-api-access-q5cq9\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:47.098604 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:47.098418 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9543233d-9115-4459-87b7-dea47aa8d1ca-istio-token\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:47.216399 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:47.216296 4292 patch_prober.go:29] interesting pod/e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Oct 30 20:16:47.216889 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:47.216394 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn" podUID="a4f584f0-f6b6-474d-ba63-967f27c5a8ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Oct 30 20:16:47.416290 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:47.416260 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp"] Oct 30 20:16:47.420672 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:47.420647 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-f53eb9-openshift-default-6fb6597977-76thp"] Oct 30 20:16:48.127704 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:48.127670 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k"] Oct 30 20:16:48.128183 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:48.128165 4292 cpu_manager.go:414] "containerMap: removing container" podUID="9543233d-9115-4459-87b7-dea47aa8d1ca" containerName="istio-proxy" Oct 30 20:16:48.128285 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:48.128185 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9543233d-9115-4459-87b7-dea47aa8d1ca" containerName="istio-proxy" Oct 30 20:16:48.128285 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:48.128224 4292 cpu_manager.go:414] "containerMap: removing container" podUID="9543233d-9115-4459-87b7-dea47aa8d1ca" containerName="istio-proxy" Oct 30 20:16:48.128285 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:48.128232 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9543233d-9115-4459-87b7-dea47aa8d1ca" containerName="istio-proxy" Oct 30 20:16:48.128506 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:48.128332 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="9543233d-9115-4459-87b7-dea47aa8d1ca" containerName="istio-proxy" Oct 30 20:16:48.132961 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:48.132941 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k" Oct 30 20:16:48.210144 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:48.210103 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ea3800f-f814-4036-b3f4-b7e3573c011b-istio-envoy\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k\" (UID: \"4ea3800f-f814-4036-b3f4-b7e3573c011b\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k" Oct 30 20:16:48.210650 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:48.210188 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ea3800f-f814-4036-b3f4-b7e3573c011b-istio-data\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k\" (UID: \"4ea3800f-f814-4036-b3f4-b7e3573c011b\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k" Oct 30 20:16:48.210650 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:48.210222 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ea3800f-f814-4036-b3f4-b7e3573c011b-istiod-ca-cert\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k\" (UID: \"4ea3800f-f814-4036-b3f4-b7e3573c011b\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k" Oct 30 20:16:48.210650 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:48.210308 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ea3800f-f814-4036-b3f4-b7e3573c011b-credential-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k\" (UID: \"4ea3800f-f814-4036-b3f4-b7e3573c011b\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k" Oct 30 20:16:48.210650 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:48.210341 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ea3800f-f814-4036-b3f4-b7e3573c011b-workload-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k\" (UID: \"4ea3800f-f814-4036-b3f4-b7e3573c011b\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k" Oct 30 20:16:48.210650 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:48.210425 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-565ht\" (UniqueName: \"kubernetes.io/projected/4ea3800f-f814-4036-b3f4-b7e3573c011b-kube-api-access-565ht\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k\" (UID: \"4ea3800f-f814-4036-b3f4-b7e3573c011b\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k" Oct 30 20:16:48.210650 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:48.210499 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ea3800f-f814-4036-b3f4-b7e3573c011b-istio-token\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k\" (UID: \"4ea3800f-f814-4036-b3f4-b7e3573c011b\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k" Oct 30 20:16:48.210650 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:48.210530 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ea3800f-f814-4036-b3f4-b7e3573c011b-workload-certs\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k\" (UID: \"4ea3800f-f814-4036-b3f4-b7e3573c011b\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k" Oct 30 20:16:48.210650 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:48.210571 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ea3800f-f814-4036-b3f4-b7e3573c011b-istio-podinfo\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k\" (UID: \"4ea3800f-f814-4036-b3f4-b7e3573c011b\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k" Oct 30 20:16:48.215921 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:48.215894 4292 patch_prober.go:29] interesting pod/e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Oct 30 20:16:48.216081 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:48.216058 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn" podUID="a4f584f0-f6b6-474d-ba63-967f27c5a8ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Oct 30 20:16:48.217405 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:48.217381 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k"] Oct 30 20:16:48.311818 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:48.311787 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ea3800f-f814-4036-b3f4-b7e3573c011b-istio-token\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k\" (UID: \"4ea3800f-f814-4036-b3f4-b7e3573c011b\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k" Oct 30 20:16:48.312043 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:48.312029 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ea3800f-f814-4036-b3f4-b7e3573c011b-workload-certs\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k\" (UID: \"4ea3800f-f814-4036-b3f4-b7e3573c011b\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k" Oct 30 20:16:48.312152 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:48.312141 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ea3800f-f814-4036-b3f4-b7e3573c011b-istio-podinfo\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k\" (UID: \"4ea3800f-f814-4036-b3f4-b7e3573c011b\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k" Oct 30 20:16:48.312248 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:48.312236 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ea3800f-f814-4036-b3f4-b7e3573c011b-istio-envoy\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k\" (UID: \"4ea3800f-f814-4036-b3f4-b7e3573c011b\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k" Oct 30 20:16:48.312379 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:48.312362 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ea3800f-f814-4036-b3f4-b7e3573c011b-istio-data\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k\" (UID: \"4ea3800f-f814-4036-b3f4-b7e3573c011b\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k" Oct 30 20:16:48.312522 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:48.312503 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ea3800f-f814-4036-b3f4-b7e3573c011b-istiod-ca-cert\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k\" (UID: \"4ea3800f-f814-4036-b3f4-b7e3573c011b\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k" Oct 30 20:16:48.312616 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:48.312415 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ea3800f-f814-4036-b3f4-b7e3573c011b-workload-certs\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k\" (UID: \"4ea3800f-f814-4036-b3f4-b7e3573c011b\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k" Oct 30 20:16:48.313184 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:48.312681 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ea3800f-f814-4036-b3f4-b7e3573c011b-istio-data\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k\" (UID: \"4ea3800f-f814-4036-b3f4-b7e3573c011b\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k" Oct 30 20:16:48.313184 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:48.312768 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ea3800f-f814-4036-b3f4-b7e3573c011b-credential-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k\" (UID: \"4ea3800f-f814-4036-b3f4-b7e3573c011b\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k" Oct 30 20:16:48.313184 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:48.312802 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ea3800f-f814-4036-b3f4-b7e3573c011b-workload-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k\" (UID: \"4ea3800f-f814-4036-b3f4-b7e3573c011b\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k" Oct 30 20:16:48.313184 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:48.312861 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-565ht\" (UniqueName: \"kubernetes.io/projected/4ea3800f-f814-4036-b3f4-b7e3573c011b-kube-api-access-565ht\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k\" (UID: \"4ea3800f-f814-4036-b3f4-b7e3573c011b\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k" Oct 30 20:16:48.313184 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:48.313002 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ea3800f-f814-4036-b3f4-b7e3573c011b-credential-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k\" (UID: \"4ea3800f-f814-4036-b3f4-b7e3573c011b\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k" Oct 30 20:16:48.313184 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:48.313169 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ea3800f-f814-4036-b3f4-b7e3573c011b-workload-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k\" (UID: \"4ea3800f-f814-4036-b3f4-b7e3573c011b\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k" Oct 30 20:16:48.313584 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:48.313347 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ea3800f-f814-4036-b3f4-b7e3573c011b-istiod-ca-cert\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k\" (UID: \"4ea3800f-f814-4036-b3f4-b7e3573c011b\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k" Oct 30 20:16:48.314537 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:48.314513 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ea3800f-f814-4036-b3f4-b7e3573c011b-istio-envoy\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k\" (UID: \"4ea3800f-f814-4036-b3f4-b7e3573c011b\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k" Oct 30 20:16:48.315097 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:48.315076 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ea3800f-f814-4036-b3f4-b7e3573c011b-istio-podinfo\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k\" (UID: \"4ea3800f-f814-4036-b3f4-b7e3573c011b\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k" Oct 30 20:16:48.377212 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:48.377174 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ea3800f-f814-4036-b3f4-b7e3573c011b-istio-token\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k\" (UID: \"4ea3800f-f814-4036-b3f4-b7e3573c011b\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k" Oct 30 20:16:48.377407 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:48.377293 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-565ht\" (UniqueName: \"kubernetes.io/projected/4ea3800f-f814-4036-b3f4-b7e3573c011b-kube-api-access-565ht\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k\" (UID: \"4ea3800f-f814-4036-b3f4-b7e3573c011b\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k" Oct 30 20:16:48.446321 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:48.446231 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k" Oct 30 20:16:48.588494 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:48.588460 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9543233d-9115-4459-87b7-dea47aa8d1ca" path="/var/lib/kubelet/pods/9543233d-9115-4459-87b7-dea47aa8d1ca/volumes" Oct 30 20:16:48.908340 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:48.908311 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k"] Oct 30 20:16:48.909900 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:16:48.909875 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ea3800f_f814_4036_b3f4_b7e3573c011b.slice/crio-5584dd58fea5014e2d83638d8390f90f96977e165471ea54a68deff5538c0f9d WatchSource:0}: Error finding container 5584dd58fea5014e2d83638d8390f90f96977e165471ea54a68deff5538c0f9d: Status 404 returned error can't find the container with id 5584dd58fea5014e2d83638d8390f90f96977e165471ea54a68deff5538c0f9d Oct 30 20:16:48.912511 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:48.912484 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:16:48.912580 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:48.912545 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:16:48.912580 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:48.912573 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:16:49.006556 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:49.006523 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k" event={"ID":"4ea3800f-f814-4036-b3f4-b7e3573c011b","Type":"ContainerStarted","Data":"3242952e1de0e1fa818747fa2bf87de56695f5622a5361187ad7dade0afc32fa"} Oct 30 20:16:49.006715 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:49.006579 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k" event={"ID":"4ea3800f-f814-4036-b3f4-b7e3573c011b","Type":"ContainerStarted","Data":"5584dd58fea5014e2d83638d8390f90f96977e165471ea54a68deff5538c0f9d"} Oct 30 20:16:49.135006 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:49.134941 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k" podStartSLOduration=1.134925676 podStartE2EDuration="1.134925676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:49.130807044 +0000 UTC m=+3112.649122498" watchObservedRunningTime="2026-10-30 20:16:49.134925676 +0000 UTC m=+3112.653241133" Oct 30 20:16:49.216215 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:49.216122 4292 patch_prober.go:29] interesting pod/e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Oct 30 20:16:49.216215 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:49.216189 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn" podUID="a4f584f0-f6b6-474d-ba63-967f27c5a8ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Oct 30 20:16:49.447941 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:49.447914 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k" Oct 30 20:16:49.448432 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:49.448117 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Oct 30 20:16:49.448568 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:49.448551 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k" podUID="4ea3800f-f814-4036-b3f4-b7e3573c011b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Oct 30 20:16:50.217916 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:50.217881 4292 patch_prober.go:29] interesting pod/e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Oct 30 20:16:50.218246 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:50.218198 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn" podUID="a4f584f0-f6b6-474d-ba63-967f27c5a8ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Oct 30 20:16:50.447070 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:50.447039 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Oct 30 20:16:50.447288 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:50.447265 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k" podUID="4ea3800f-f814-4036-b3f4-b7e3573c011b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Oct 30 20:16:51.216120 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:51.216070 4292 patch_prober.go:29] interesting pod/e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Oct 30 20:16:51.216692 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:51.216128 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn" podUID="a4f584f0-f6b6-474d-ba63-967f27c5a8ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Oct 30 20:16:51.447407 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:51.447344 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Oct 30 20:16:51.447584 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:51.447414 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k" podUID="4ea3800f-f814-4036-b3f4-b7e3573c011b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Oct 30 20:16:52.456375 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:52.454497 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Oct 30 20:16:52.456375 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:52.454562 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k" podUID="4ea3800f-f814-4036-b3f4-b7e3573c011b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Oct 30 20:16:52.752591 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:52.752504 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn" Oct 30 20:16:52.753771 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:52.753743 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn" Oct 30 20:16:52.760585 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:52.760562 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn" Oct 30 20:16:53.447452 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:53.447418 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Oct 30 20:16:53.447631 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:53.447469 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k" podUID="4ea3800f-f814-4036-b3f4-b7e3573c011b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Oct 30 20:16:54.447609 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:54.447578 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Oct 30 20:16:54.448014 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:54.447646 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k" podUID="4ea3800f-f814-4036-b3f4-b7e3573c011b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Oct 30 20:16:55.446601 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:55.446566 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Oct 30 20:16:55.446813 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:55.446625 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k" podUID="4ea3800f-f814-4036-b3f4-b7e3573c011b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Oct 30 20:16:56.495859 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:56.495827 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k" Oct 30 20:16:56.496416 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:56.496392 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k" Oct 30 20:16:56.497233 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:56.497214 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k" Oct 30 20:16:57.400048 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:57.400018 4292 scope.go:122] "RemoveContainer" containerID="a2c9d8f3c9d550b70e7cb7b2a6deb74c436514662d21a94beec572f7f232c0df" Oct 30 20:16:57.411783 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:16:57.411758 4292 scope.go:122] "RemoveContainer" containerID="f205f99f62955a5e9b782fe3798cd80efc689feea87b0e956e19c94a7feb4987" Oct 30 20:17:01.711648 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:01.711617 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p"] Oct 30 20:17:01.712181 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:01.712167 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="9543233d-9115-4459-87b7-dea47aa8d1ca" containerName="istio-proxy" Oct 30 20:17:01.715098 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:01.715079 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p" Oct 30 20:17:01.768826 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:01.768789 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-bbfa07-openshift-default-dockercfg-vj7kd\"" Oct 30 20:17:01.833652 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:01.833618 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p"] Oct 30 20:17:01.863793 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:01.862926 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb07cea-63a9-4744-9ce5-fd802f091a82-workload-socket\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p\" (UID: \"1fb07cea-63a9-4744-9ce5-fd802f091a82\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p" Oct 30 20:17:01.863793 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:01.863043 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb07cea-63a9-4744-9ce5-fd802f091a82-credential-socket\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p\" (UID: \"1fb07cea-63a9-4744-9ce5-fd802f091a82\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p" Oct 30 20:17:01.863793 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:01.863083 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fb07cea-63a9-4744-9ce5-fd802f091a82-istiod-ca-cert\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p\" (UID: \"1fb07cea-63a9-4744-9ce5-fd802f091a82\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p" Oct 30 20:17:01.863793 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:01.863166 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fb07cea-63a9-4744-9ce5-fd802f091a82-istio-data\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p\" (UID: \"1fb07cea-63a9-4744-9ce5-fd802f091a82\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p" Oct 30 20:17:01.863793 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:01.863208 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fb07cea-63a9-4744-9ce5-fd802f091a82-istio-podinfo\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p\" (UID: \"1fb07cea-63a9-4744-9ce5-fd802f091a82\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p" Oct 30 20:17:01.863793 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:01.863268 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fb07cea-63a9-4744-9ce5-fd802f091a82-istio-envoy\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p\" (UID: \"1fb07cea-63a9-4744-9ce5-fd802f091a82\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p" Oct 30 20:17:01.863793 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:01.863338 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fb07cea-63a9-4744-9ce5-fd802f091a82-istio-token\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p\" (UID: \"1fb07cea-63a9-4744-9ce5-fd802f091a82\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p" Oct 30 20:17:01.863793 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:01.863383 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fb07cea-63a9-4744-9ce5-fd802f091a82-workload-certs\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p\" (UID: \"1fb07cea-63a9-4744-9ce5-fd802f091a82\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p" Oct 30 20:17:01.863793 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:01.863420 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq2ds\" (UniqueName: \"kubernetes.io/projected/1fb07cea-63a9-4744-9ce5-fd802f091a82-kube-api-access-sq2ds\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p\" (UID: \"1fb07cea-63a9-4744-9ce5-fd802f091a82\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p" Oct 30 20:17:01.965132 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:01.965046 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fb07cea-63a9-4744-9ce5-fd802f091a82-istio-data\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p\" (UID: \"1fb07cea-63a9-4744-9ce5-fd802f091a82\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p" Oct 30 20:17:01.965132 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:01.965086 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fb07cea-63a9-4744-9ce5-fd802f091a82-istio-podinfo\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p\" (UID: \"1fb07cea-63a9-4744-9ce5-fd802f091a82\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p" Oct 30 20:17:01.965132 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:01.965114 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fb07cea-63a9-4744-9ce5-fd802f091a82-istio-envoy\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p\" (UID: \"1fb07cea-63a9-4744-9ce5-fd802f091a82\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p" Oct 30 20:17:01.965417 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:01.965160 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fb07cea-63a9-4744-9ce5-fd802f091a82-istio-token\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p\" (UID: \"1fb07cea-63a9-4744-9ce5-fd802f091a82\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p" Oct 30 20:17:01.965417 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:01.965185 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fb07cea-63a9-4744-9ce5-fd802f091a82-workload-certs\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p\" (UID: \"1fb07cea-63a9-4744-9ce5-fd802f091a82\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p" Oct 30 20:17:01.965417 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:01.965210 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq2ds\" (UniqueName: \"kubernetes.io/projected/1fb07cea-63a9-4744-9ce5-fd802f091a82-kube-api-access-sq2ds\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p\" (UID: \"1fb07cea-63a9-4744-9ce5-fd802f091a82\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p" Oct 30 20:17:01.965417 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:01.965283 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb07cea-63a9-4744-9ce5-fd802f091a82-workload-socket\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p\" (UID: \"1fb07cea-63a9-4744-9ce5-fd802f091a82\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p" Oct 30 20:17:01.965417 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:01.965388 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb07cea-63a9-4744-9ce5-fd802f091a82-credential-socket\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p\" (UID: \"1fb07cea-63a9-4744-9ce5-fd802f091a82\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p" Oct 30 20:17:01.965725 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:01.965421 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fb07cea-63a9-4744-9ce5-fd802f091a82-istiod-ca-cert\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p\" (UID: \"1fb07cea-63a9-4744-9ce5-fd802f091a82\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p" Oct 30 20:17:01.965725 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:01.965552 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fb07cea-63a9-4744-9ce5-fd802f091a82-istio-data\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p\" (UID: \"1fb07cea-63a9-4744-9ce5-fd802f091a82\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p" Oct 30 20:17:01.965725 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:01.965653 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fb07cea-63a9-4744-9ce5-fd802f091a82-workload-certs\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p\" (UID: \"1fb07cea-63a9-4744-9ce5-fd802f091a82\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p" Oct 30 20:17:01.965846 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:01.965779 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb07cea-63a9-4744-9ce5-fd802f091a82-workload-socket\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p\" (UID: \"1fb07cea-63a9-4744-9ce5-fd802f091a82\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p" Oct 30 20:17:01.965846 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:01.965836 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb07cea-63a9-4744-9ce5-fd802f091a82-credential-socket\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p\" (UID: \"1fb07cea-63a9-4744-9ce5-fd802f091a82\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p" Oct 30 20:17:01.966241 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:01.966214 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fb07cea-63a9-4744-9ce5-fd802f091a82-istiod-ca-cert\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p\" (UID: \"1fb07cea-63a9-4744-9ce5-fd802f091a82\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p" Oct 30 20:17:01.967533 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:01.967510 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fb07cea-63a9-4744-9ce5-fd802f091a82-istio-envoy\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p\" (UID: \"1fb07cea-63a9-4744-9ce5-fd802f091a82\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p" Oct 30 20:17:01.967733 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:01.967712 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fb07cea-63a9-4744-9ce5-fd802f091a82-istio-podinfo\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p\" (UID: \"1fb07cea-63a9-4744-9ce5-fd802f091a82\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p" Oct 30 20:17:02.027405 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:02.027373 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fb07cea-63a9-4744-9ce5-fd802f091a82-istio-token\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p\" (UID: \"1fb07cea-63a9-4744-9ce5-fd802f091a82\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p" Oct 30 20:17:02.027552 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:02.027443 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq2ds\" (UniqueName: \"kubernetes.io/projected/1fb07cea-63a9-4744-9ce5-fd802f091a82-kube-api-access-sq2ds\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p\" (UID: \"1fb07cea-63a9-4744-9ce5-fd802f091a82\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p" Oct 30 20:17:02.027623 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:02.027564 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p" Oct 30 20:17:02.460965 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:02.460940 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p"] Oct 30 20:17:02.462072 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:17:02.462046 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fb07cea_63a9_4744_9ce5_fd802f091a82.slice/crio-72508a584494b7eb340dae8f96f515084feb4b0bab3ea9f60c16510fa10c1bdd WatchSource:0}: Error finding container 72508a584494b7eb340dae8f96f515084feb4b0bab3ea9f60c16510fa10c1bdd: Status 404 returned error can't find the container with id 72508a584494b7eb340dae8f96f515084feb4b0bab3ea9f60c16510fa10c1bdd Oct 30 20:17:02.464605 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:02.464564 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:17:02.464715 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:02.464639 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:17:02.464715 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:02.464672 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:17:03.180986 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:03.180948 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p" event={"ID":"1fb07cea-63a9-4744-9ce5-fd802f091a82","Type":"ContainerStarted","Data":"9fef014fa514f89fef83336cbeb3e719def15ef94762276d678a9db4dceba41c"} Oct 30 20:17:03.181513 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:03.181001 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p" event={"ID":"1fb07cea-63a9-4744-9ce5-fd802f091a82","Type":"ContainerStarted","Data":"72508a584494b7eb340dae8f96f515084feb4b0bab3ea9f60c16510fa10c1bdd"} Oct 30 20:17:03.338158 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:03.338102 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p" podStartSLOduration=2.338086916 podStartE2EDuration="2.338086916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:17:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:17:03.337242567 +0000 UTC m=+3126.855558025" watchObservedRunningTime="2026-10-30 20:17:03.338086916 +0000 UTC m=+3126.856402372" Oct 30 20:17:04.027863 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:04.027818 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p" Oct 30 20:17:04.029863 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:04.029835 4292 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 20:17:04.030003 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:04.029885 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p" podUID="1fb07cea-63a9-4744-9ce5-fd802f091a82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 20:17:05.028783 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:05.028745 4292 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 20:17:05.029181 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:05.028800 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p" podUID="1fb07cea-63a9-4744-9ce5-fd802f091a82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 20:17:06.027900 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:06.027866 4292 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 20:17:06.028100 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:06.027918 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p" podUID="1fb07cea-63a9-4744-9ce5-fd802f091a82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 20:17:07.027963 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:07.027920 4292 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 20:17:07.028407 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:07.027979 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p" podUID="1fb07cea-63a9-4744-9ce5-fd802f091a82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 20:17:08.028697 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:08.028662 4292 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 20:17:08.029120 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:08.028756 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p" podUID="1fb07cea-63a9-4744-9ce5-fd802f091a82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 20:17:09.028456 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:09.028415 4292 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 20:17:09.028668 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:09.028502 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p" podUID="1fb07cea-63a9-4744-9ce5-fd802f091a82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 20:17:10.032441 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:10.032409 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p" Oct 30 20:17:10.032934 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:10.032915 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p" Oct 30 20:17:10.033728 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:10.033706 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p" Oct 30 20:17:13.220448 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:17:13.220400 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:17:16.835437 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:16.835396 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb"] Oct 30 20:17:16.840478 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:16.840454 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb" Oct 30 20:17:16.939510 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:16.939483 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/494c9125-ea0a-4656-af2b-f0d198271ef3-istiod-ca-cert\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb\" (UID: \"494c9125-ea0a-4656-af2b-f0d198271ef3\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb" Oct 30 20:17:16.939693 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:16.939532 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/494c9125-ea0a-4656-af2b-f0d198271ef3-credential-socket\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb\" (UID: \"494c9125-ea0a-4656-af2b-f0d198271ef3\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb" Oct 30 20:17:16.939693 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:16.939553 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/494c9125-ea0a-4656-af2b-f0d198271ef3-istio-token\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb\" (UID: \"494c9125-ea0a-4656-af2b-f0d198271ef3\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb" Oct 30 20:17:16.939693 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:16.939654 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/494c9125-ea0a-4656-af2b-f0d198271ef3-istio-podinfo\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb\" (UID: \"494c9125-ea0a-4656-af2b-f0d198271ef3\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb" Oct 30 20:17:16.939859 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:16.939703 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/494c9125-ea0a-4656-af2b-f0d198271ef3-workload-certs\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb\" (UID: \"494c9125-ea0a-4656-af2b-f0d198271ef3\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb" Oct 30 20:17:16.939859 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:16.939721 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/494c9125-ea0a-4656-af2b-f0d198271ef3-istio-data\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb\" (UID: \"494c9125-ea0a-4656-af2b-f0d198271ef3\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb" Oct 30 20:17:16.939859 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:16.939830 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g727q\" (UniqueName: \"kubernetes.io/projected/494c9125-ea0a-4656-af2b-f0d198271ef3-kube-api-access-g727q\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb\" (UID: \"494c9125-ea0a-4656-af2b-f0d198271ef3\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb" Oct 30 20:17:16.940018 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:16.939959 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/494c9125-ea0a-4656-af2b-f0d198271ef3-workload-socket\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb\" (UID: \"494c9125-ea0a-4656-af2b-f0d198271ef3\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb" Oct 30 20:17:16.940018 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:16.939995 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/494c9125-ea0a-4656-af2b-f0d198271ef3-istio-envoy\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb\" (UID: \"494c9125-ea0a-4656-af2b-f0d198271ef3\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb" Oct 30 20:17:16.948581 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:16.948551 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb"] Oct 30 20:17:17.041656 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:17.041620 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g727q\" (UniqueName: \"kubernetes.io/projected/494c9125-ea0a-4656-af2b-f0d198271ef3-kube-api-access-g727q\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb\" (UID: \"494c9125-ea0a-4656-af2b-f0d198271ef3\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb" Oct 30 20:17:17.041845 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:17.041699 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/494c9125-ea0a-4656-af2b-f0d198271ef3-workload-socket\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb\" (UID: \"494c9125-ea0a-4656-af2b-f0d198271ef3\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb" Oct 30 20:17:17.041845 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:17.041727 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/494c9125-ea0a-4656-af2b-f0d198271ef3-istio-envoy\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb\" (UID: \"494c9125-ea0a-4656-af2b-f0d198271ef3\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb" Oct 30 20:17:17.041845 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:17.041776 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/494c9125-ea0a-4656-af2b-f0d198271ef3-istiod-ca-cert\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb\" (UID: \"494c9125-ea0a-4656-af2b-f0d198271ef3\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb" Oct 30 20:17:17.041845 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:17.041812 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/494c9125-ea0a-4656-af2b-f0d198271ef3-credential-socket\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb\" (UID: \"494c9125-ea0a-4656-af2b-f0d198271ef3\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb" Oct 30 20:17:17.041845 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:17.041836 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/494c9125-ea0a-4656-af2b-f0d198271ef3-istio-token\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb\" (UID: \"494c9125-ea0a-4656-af2b-f0d198271ef3\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb" Oct 30 20:17:17.042097 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:17.041975 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/494c9125-ea0a-4656-af2b-f0d198271ef3-istio-podinfo\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb\" (UID: \"494c9125-ea0a-4656-af2b-f0d198271ef3\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb" Oct 30 20:17:17.042097 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:17.042068 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/494c9125-ea0a-4656-af2b-f0d198271ef3-workload-certs\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb\" (UID: \"494c9125-ea0a-4656-af2b-f0d198271ef3\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb" Oct 30 20:17:17.042097 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:17.042091 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/494c9125-ea0a-4656-af2b-f0d198271ef3-istio-data\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb\" (UID: \"494c9125-ea0a-4656-af2b-f0d198271ef3\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb" Oct 30 20:17:17.042507 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:17.042167 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/494c9125-ea0a-4656-af2b-f0d198271ef3-workload-socket\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb\" (UID: \"494c9125-ea0a-4656-af2b-f0d198271ef3\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb" Oct 30 20:17:17.042507 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:17.042404 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/494c9125-ea0a-4656-af2b-f0d198271ef3-workload-certs\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb\" (UID: \"494c9125-ea0a-4656-af2b-f0d198271ef3\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb" Oct 30 20:17:17.042654 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:17.042571 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/494c9125-ea0a-4656-af2b-f0d198271ef3-istiod-ca-cert\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb\" (UID: \"494c9125-ea0a-4656-af2b-f0d198271ef3\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb" Oct 30 20:17:17.042737 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:17.042715 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/494c9125-ea0a-4656-af2b-f0d198271ef3-credential-socket\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb\" (UID: \"494c9125-ea0a-4656-af2b-f0d198271ef3\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb" Oct 30 20:17:17.042796 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:17.042721 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/494c9125-ea0a-4656-af2b-f0d198271ef3-istio-data\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb\" (UID: \"494c9125-ea0a-4656-af2b-f0d198271ef3\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb" Oct 30 20:17:17.044614 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:17.044589 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/494c9125-ea0a-4656-af2b-f0d198271ef3-istio-envoy\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb\" (UID: \"494c9125-ea0a-4656-af2b-f0d198271ef3\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb" Oct 30 20:17:17.044815 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:17.044797 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/494c9125-ea0a-4656-af2b-f0d198271ef3-istio-podinfo\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb\" (UID: \"494c9125-ea0a-4656-af2b-f0d198271ef3\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb" Oct 30 20:17:17.104522 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:17.104410 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/494c9125-ea0a-4656-af2b-f0d198271ef3-istio-token\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb\" (UID: \"494c9125-ea0a-4656-af2b-f0d198271ef3\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb" Oct 30 20:17:17.104522 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:17.104483 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g727q\" (UniqueName: \"kubernetes.io/projected/494c9125-ea0a-4656-af2b-f0d198271ef3-kube-api-access-g727q\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb\" (UID: \"494c9125-ea0a-4656-af2b-f0d198271ef3\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb" Oct 30 20:17:17.168890 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:17.168576 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb" Oct 30 20:17:17.605525 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:17.605495 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb"] Oct 30 20:17:17.606972 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:17:17.606944 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod494c9125_ea0a_4656_af2b_f0d198271ef3.slice/crio-c54d7af12437dd9cb95f03b13f26e7eb864032ab40bc84803d093b25f970813f WatchSource:0}: Error finding container c54d7af12437dd9cb95f03b13f26e7eb864032ab40bc84803d093b25f970813f: Status 404 returned error can't find the container with id c54d7af12437dd9cb95f03b13f26e7eb864032ab40bc84803d093b25f970813f Oct 30 20:17:17.625856 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:17.610235 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:17:17.625856 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:17.610426 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:17:17.625856 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:17.610697 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:17:18.296746 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:18.296703 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb" event={"ID":"494c9125-ea0a-4656-af2b-f0d198271ef3","Type":"ContainerStarted","Data":"a1fb257a32adaa69638f0a9ce7cefd59ea1be8bc6c0594bbd7725b9f71149fcb"} Oct 30 20:17:18.297211 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:18.296802 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb" event={"ID":"494c9125-ea0a-4656-af2b-f0d198271ef3","Type":"ContainerStarted","Data":"c54d7af12437dd9cb95f03b13f26e7eb864032ab40bc84803d093b25f970813f"} Oct 30 20:17:18.440188 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:18.440128 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb" podStartSLOduration=2.440108998 podStartE2EDuration="2.440108998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:17:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:17:18.436755668 +0000 UTC m=+3141.955071143" watchObservedRunningTime="2026-10-30 20:17:18.440108998 +0000 UTC m=+3141.958424456" Oct 30 20:17:19.171008 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:19.170976 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb" Oct 30 20:17:19.171443 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:19.171414 4292 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Oct 30 20:17:19.171637 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:19.171614 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb" podUID="494c9125-ea0a-4656-af2b-f0d198271ef3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Oct 30 20:17:20.173376 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:20.170787 4292 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Oct 30 20:17:20.173376 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:20.170851 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb" podUID="494c9125-ea0a-4656-af2b-f0d198271ef3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Oct 30 20:17:21.174968 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:21.174887 4292 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Oct 30 20:17:21.174968 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:21.174944 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb" podUID="494c9125-ea0a-4656-af2b-f0d198271ef3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Oct 30 20:17:22.174806 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:22.174768 4292 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Oct 30 20:17:22.175000 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:22.174837 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb" podUID="494c9125-ea0a-4656-af2b-f0d198271ef3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Oct 30 20:17:22.270374 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:22.270015 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d5d3ba-cwxq7"] Oct 30 20:17:22.281144 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:22.281108 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d5d3ba-cwxq7" Oct 30 20:17:22.304692 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:22.304614 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd8ce91a-db85-4d15-9ad0-738afb4a092a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-d5d3ba-cwxq7\" (UID: \"dd8ce91a-db85-4d15-9ad0-738afb4a092a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d5d3ba-cwxq7" Oct 30 20:17:22.304692 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:22.304663 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd8ce91a-db85-4d15-9ad0-738afb4a092a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-d5d3ba-cwxq7\" (UID: \"dd8ce91a-db85-4d15-9ad0-738afb4a092a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d5d3ba-cwxq7" Oct 30 20:17:22.304942 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:22.304860 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdm49\" (UniqueName: \"kubernetes.io/projected/dd8ce91a-db85-4d15-9ad0-738afb4a092a-kube-api-access-wdm49\") pod \"maas-api-revoke-keys-e2e-praxis-a-d5d3ba-cwxq7\" (UID: \"dd8ce91a-db85-4d15-9ad0-738afb4a092a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d5d3ba-cwxq7" Oct 30 20:17:22.313240 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:22.313192 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d5d3ba-cwxq7"] Oct 30 20:17:22.334786 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:22.334748 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:17:22.374391 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:22.361535 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\"" Oct 30 20:17:22.405751 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:22.405707 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd8ce91a-db85-4d15-9ad0-738afb4a092a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-d5d3ba-cwxq7\" (UID: \"dd8ce91a-db85-4d15-9ad0-738afb4a092a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d5d3ba-cwxq7" Oct 30 20:17:22.405954 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:22.405766 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd8ce91a-db85-4d15-9ad0-738afb4a092a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-d5d3ba-cwxq7\" (UID: \"dd8ce91a-db85-4d15-9ad0-738afb4a092a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d5d3ba-cwxq7" Oct 30 20:17:22.406031 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:22.405954 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdm49\" (UniqueName: \"kubernetes.io/projected/dd8ce91a-db85-4d15-9ad0-738afb4a092a-kube-api-access-wdm49\") pod \"maas-api-revoke-keys-e2e-praxis-a-d5d3ba-cwxq7\" (UID: \"dd8ce91a-db85-4d15-9ad0-738afb4a092a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d5d3ba-cwxq7" Oct 30 20:17:22.406339 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:22.406315 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd8ce91a-db85-4d15-9ad0-738afb4a092a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-d5d3ba-cwxq7\" (UID: \"dd8ce91a-db85-4d15-9ad0-738afb4a092a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d5d3ba-cwxq7" Oct 30 20:17:22.408309 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:22.408286 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd8ce91a-db85-4d15-9ad0-738afb4a092a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-d5d3ba-cwxq7\" (UID: \"dd8ce91a-db85-4d15-9ad0-738afb4a092a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d5d3ba-cwxq7" Oct 30 20:17:22.475154 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:22.475075 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdm49\" (UniqueName: \"kubernetes.io/projected/dd8ce91a-db85-4d15-9ad0-738afb4a092a-kube-api-access-wdm49\") pod \"maas-api-revoke-keys-e2e-praxis-a-d5d3ba-cwxq7\" (UID: \"dd8ce91a-db85-4d15-9ad0-738afb4a092a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d5d3ba-cwxq7" Oct 30 20:17:22.596593 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:22.596561 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d5d3ba-cwxq7" Oct 30 20:17:22.992262 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:22.992206 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d5d3ba-cwxq7"] Oct 30 20:17:22.995496 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:17:22.995463 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd8ce91a_db85_4d15_9ad0_738afb4a092a.slice/crio-3ecea27bf408346f76197c643d4531393d46f14e59ec42fbf48978da4442918d WatchSource:0}: Error finding container 3ecea27bf408346f76197c643d4531393d46f14e59ec42fbf48978da4442918d: Status 404 returned error can't find the container with id 3ecea27bf408346f76197c643d4531393d46f14e59ec42fbf48978da4442918d Oct 30 20:17:23.177685 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:23.177203 4292 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Oct 30 20:17:23.177685 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:23.177260 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb" podUID="494c9125-ea0a-4656-af2b-f0d198271ef3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Oct 30 20:17:23.335037 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:23.334995 4292 generic.go:356] "Generic (PLEG): container finished" podID="f28646cd-7d71-45f8-9a70-407819e9b293" containerID="814c4b7f8c40ba7416ba75b6addbf79d8f51ba597a7a28de3bdfe43d459c4208" exitCode=0 Oct 30 20:17:23.335195 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:23.335054 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj" event={"ID":"f28646cd-7d71-45f8-9a70-407819e9b293","Type":"ContainerDied","Data":"814c4b7f8c40ba7416ba75b6addbf79d8f51ba597a7a28de3bdfe43d459c4208"} Oct 30 20:17:23.335690 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:23.335670 4292 scope.go:122] "RemoveContainer" containerID="814c4b7f8c40ba7416ba75b6addbf79d8f51ba597a7a28de3bdfe43d459c4208" Oct 30 20:17:23.336639 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:23.336616 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d5d3ba-cwxq7" event={"ID":"dd8ce91a-db85-4d15-9ad0-738afb4a092a","Type":"ContainerStarted","Data":"3ecea27bf408346f76197c643d4531393d46f14e59ec42fbf48978da4442918d"} Oct 30 20:17:23.337269 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:23.337239 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:17:23.337395 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:23.337320 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:17:23.337469 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:23.337405 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:17:24.169693 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:24.169658 4292 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Oct 30 20:17:24.169885 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:24.169716 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb" podUID="494c9125-ea0a-4656-af2b-f0d198271ef3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Oct 30 20:17:24.348813 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:24.348777 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj" event={"ID":"f28646cd-7d71-45f8-9a70-407819e9b293","Type":"ContainerStarted","Data":"b48c5601c5ab92d1034c2bf947365222a8e19aabb725e9e795a72501bcc80815"} Oct 30 20:17:24.351209 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:24.350857 4292 generic.go:356] "Generic (PLEG): container finished" podID="dd8ce91a-db85-4d15-9ad0-738afb4a092a" containerID="16240c9b18721fcefcbd1bb57e9fa28872555bc5b6f241536d95b094dc941b9b" exitCode=0 Oct 30 20:17:24.351209 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:24.351053 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d5d3ba-cwxq7" event={"ID":"dd8ce91a-db85-4d15-9ad0-738afb4a092a","Type":"ContainerDied","Data":"16240c9b18721fcefcbd1bb57e9fa28872555bc5b6f241536d95b094dc941b9b"} Oct 30 20:17:24.844767 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:24.844729 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj" Oct 30 20:17:24.846363 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:24.846326 4292 patch_prober.go:29] interesting pod/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 30 20:17:24.846499 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:24.846393 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj" podUID="f28646cd-7d71-45f8-9a70-407819e9b293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 30 20:17:25.327775 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:25.327745 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb" Oct 30 20:17:25.329294 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:25.329220 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb" Oct 30 20:17:25.329720 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:25.329691 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb" Oct 30 20:17:25.588321 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:25.578522 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj" Oct 30 20:17:25.648844 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:25.648820 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d5d3ba-cwxq7" Oct 30 20:17:25.788902 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:25.788866 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd8ce91a-db85-4d15-9ad0-738afb4a092a-kube-api-access-wdm49\" (UniqueName: \"kubernetes.io/projected/dd8ce91a-db85-4d15-9ad0-738afb4a092a-kube-api-access-wdm49\") pod \"dd8ce91a-db85-4d15-9ad0-738afb4a092a\" (UID: \"dd8ce91a-db85-4d15-9ad0-738afb4a092a\") " Oct 30 20:17:25.792055 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:25.790458 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd8ce91a-db85-4d15-9ad0-738afb4a092a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd8ce91a-db85-4d15-9ad0-738afb4a092a-maas-api-xks-ca\") pod \"dd8ce91a-db85-4d15-9ad0-738afb4a092a\" (UID: \"dd8ce91a-db85-4d15-9ad0-738afb4a092a\") " Oct 30 20:17:25.792055 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:25.790520 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd8ce91a-db85-4d15-9ad0-738afb4a092a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd8ce91a-db85-4d15-9ad0-738afb4a092a-maas-api-openshift-ca\") pod \"dd8ce91a-db85-4d15-9ad0-738afb4a092a\" (UID: \"dd8ce91a-db85-4d15-9ad0-738afb4a092a\") " Oct 30 20:17:25.792055 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:25.791248 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd8ce91a-db85-4d15-9ad0-738afb4a092a-maas-api-openshift-ca" pod "dd8ce91a-db85-4d15-9ad0-738afb4a092a" (UID: "dd8ce91a-db85-4d15-9ad0-738afb4a092a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:17:25.793950 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:25.793920 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd8ce91a-db85-4d15-9ad0-738afb4a092a-kube-api-access-wdm49" pod "dd8ce91a-db85-4d15-9ad0-738afb4a092a" (UID: "dd8ce91a-db85-4d15-9ad0-738afb4a092a"). InnerVolumeSpecName "kube-api-access-wdm49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:17:25.794787 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:25.794749 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd8ce91a-db85-4d15-9ad0-738afb4a092a-maas-api-xks-ca" pod "dd8ce91a-db85-4d15-9ad0-738afb4a092a" (UID: "dd8ce91a-db85-4d15-9ad0-738afb4a092a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:17:25.845142 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:25.845061 4292 patch_prober.go:29] interesting pod/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 30 20:17:25.845142 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:25.845119 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj" podUID="f28646cd-7d71-45f8-9a70-407819e9b293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 30 20:17:25.891884 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:25.891847 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdm49\" (UniqueName: \"kubernetes.io/projected/dd8ce91a-db85-4d15-9ad0-738afb4a092a-kube-api-access-wdm49\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:25.891884 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:25.891878 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd8ce91a-db85-4d15-9ad0-738afb4a092a-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:25.891884 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:25.891889 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd8ce91a-db85-4d15-9ad0-738afb4a092a-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:26.383512 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:26.383477 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d5d3ba-cwxq7" Oct 30 20:17:26.383512 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:26.383494 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d5d3ba-cwxq7" event={"ID":"dd8ce91a-db85-4d15-9ad0-738afb4a092a","Type":"ContainerDied","Data":"3ecea27bf408346f76197c643d4531393d46f14e59ec42fbf48978da4442918d"} Oct 30 20:17:26.383735 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:26.383553 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ecea27bf408346f76197c643d4531393d46f14e59ec42fbf48978da4442918d" Oct 30 20:17:26.845037 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:26.845003 4292 patch_prober.go:29] interesting pod/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 30 20:17:26.845488 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:26.845053 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj" podUID="f28646cd-7d71-45f8-9a70-407819e9b293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 30 20:17:27.844468 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:27.844429 4292 patch_prober.go:29] interesting pod/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 30 20:17:27.844650 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:27.844487 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj" podUID="f28646cd-7d71-45f8-9a70-407819e9b293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 30 20:17:28.881373 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:28.874719 4292 patch_prober.go:29] interesting pod/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 30 20:17:28.881373 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:28.874788 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj" podUID="f28646cd-7d71-45f8-9a70-407819e9b293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 30 20:17:29.845121 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:29.845088 4292 patch_prober.go:29] interesting pod/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 30 20:17:29.845290 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:29.845174 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj" podUID="f28646cd-7d71-45f8-9a70-407819e9b293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 30 20:17:30.848958 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:30.848923 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj" Oct 30 20:17:30.850062 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:30.850039 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj" Oct 30 20:17:43.220240 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:17:43.220202 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:17:57.450174 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:57.450140 4292 scope.go:122] "RemoveContainer" containerID="a493fa2adbba3d6f10099ecdd57ac1e8cbda78618e683657a8a766c070a26216" Oct 30 20:17:57.460289 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:57.460264 4292 scope.go:122] "RemoveContainer" containerID="7481fbbc97dc3ae86fac00af8a1b639ba9b1c85047a6cd1f9f76fef350c04028" Oct 30 20:17:57.469241 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:57.469216 4292 scope.go:122] "RemoveContainer" containerID="469be6aa1928edddbba290c1c3236739608aa4a4b337049097da895e74498039" Oct 30 20:17:58.321919 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:58.321886 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d5d3ba-cwxq7"] Oct 30 20:17:58.372241 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:58.372208 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d5d3ba-cwxq7"] Oct 30 20:17:58.582463 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:17:58.582378 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd8ce91a-db85-4d15-9ad0-738afb4a092a" path="/var/lib/kubelet/pods/dd8ce91a-db85-4d15-9ad0-738afb4a092a/volumes" Oct 30 20:18:08.673872 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:08.673833 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e4864d-67698fc88c-t6ft7"] Oct 30 20:18:08.674546 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:08.674524 4292 cpu_manager.go:414] "containerMap: removing container" podUID="dd8ce91a-db85-4d15-9ad0-738afb4a092a" containerName="revoke-keys" Oct 30 20:18:08.674664 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:08.674547 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd8ce91a-db85-4d15-9ad0-738afb4a092a" containerName="revoke-keys" Oct 30 20:18:08.674718 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:08.674710 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd8ce91a-db85-4d15-9ad0-738afb4a092a" containerName="revoke-keys" Oct 30 20:18:08.678118 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:08.678095 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e4864d-67698fc88c-t6ft7" Oct 30 20:18:08.733000 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:08.732959 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e4864d-67698fc88c-t6ft7"] Oct 30 20:18:08.780046 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:08.780009 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-e4864d-dockercfg-smfz2\"" Oct 30 20:18:08.780046 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:08.780027 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-e4864d\"" Oct 30 20:18:08.832455 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:08.832412 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cab76d4c-33c4-485e-b45d-368fb2137c63-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-e4864d-67698fc88c-t6ft7\" (UID: \"cab76d4c-33c4-485e-b45d-368fb2137c63\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e4864d-67698fc88c-t6ft7" Oct 30 20:18:08.832674 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:08.832524 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2lgp\" (UniqueName: \"kubernetes.io/projected/cab76d4c-33c4-485e-b45d-368fb2137c63-kube-api-access-p2lgp\") pod \"payload-pre-processing-e2e-shared-b-w0-e4864d-67698fc88c-t6ft7\" (UID: \"cab76d4c-33c4-485e-b45d-368fb2137c63\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e4864d-67698fc88c-t6ft7" Oct 30 20:18:08.933578 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:08.933474 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2lgp\" (UniqueName: \"kubernetes.io/projected/cab76d4c-33c4-485e-b45d-368fb2137c63-kube-api-access-p2lgp\") pod \"payload-pre-processing-e2e-shared-b-w0-e4864d-67698fc88c-t6ft7\" (UID: \"cab76d4c-33c4-485e-b45d-368fb2137c63\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e4864d-67698fc88c-t6ft7" Oct 30 20:18:08.933764 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:08.933741 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cab76d4c-33c4-485e-b45d-368fb2137c63-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-e4864d-67698fc88c-t6ft7\" (UID: \"cab76d4c-33c4-485e-b45d-368fb2137c63\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e4864d-67698fc88c-t6ft7" Oct 30 20:18:08.934478 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:08.934458 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cab76d4c-33c4-485e-b45d-368fb2137c63-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-e4864d-67698fc88c-t6ft7\" (UID: \"cab76d4c-33c4-485e-b45d-368fb2137c63\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e4864d-67698fc88c-t6ft7" Oct 30 20:18:08.996419 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:08.996383 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2lgp\" (UniqueName: \"kubernetes.io/projected/cab76d4c-33c4-485e-b45d-368fb2137c63-kube-api-access-p2lgp\") pod \"payload-pre-processing-e2e-shared-b-w0-e4864d-67698fc88c-t6ft7\" (UID: \"cab76d4c-33c4-485e-b45d-368fb2137c63\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e4864d-67698fc88c-t6ft7" Oct 30 20:18:09.292189 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:09.292158 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e4864d-67698fc88c-t6ft7" Oct 30 20:18:09.674749 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:09.674716 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e4864d-67698fc88c-t6ft7"] Oct 30 20:18:09.675680 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:18:09.675653 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcab76d4c_33c4_485e_b45d_368fb2137c63.slice/crio-db9c4482c532c9f6b319f734610ef5b756cdc6410febfd11d9a6f68a4f73f703 WatchSource:0}: Error finding container db9c4482c532c9f6b319f734610ef5b756cdc6410febfd11d9a6f68a4f73f703: Status 404 returned error can't find the container with id db9c4482c532c9f6b319f734610ef5b756cdc6410febfd11d9a6f68a4f73f703 Oct 30 20:18:09.705017 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:09.704990 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e4864d-67698fc88c-t6ft7" event={"ID":"cab76d4c-33c4-485e-b45d-368fb2137c63","Type":"ContainerStarted","Data":"db9c4482c532c9f6b319f734610ef5b756cdc6410febfd11d9a6f68a4f73f703"} Oct 30 20:18:10.713089 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:10.713010 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e4864d-67698fc88c-t6ft7" event={"ID":"cab76d4c-33c4-485e-b45d-368fb2137c63","Type":"ContainerStarted","Data":"ac24575d342cab09c7b1a16638c1e18f558d2ed88148c7c31eb642230844b747"} Oct 30 20:18:10.713089 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:10.713086 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e4864d-67698fc88c-t6ft7" Oct 30 20:18:10.833997 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:10.833945 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e4864d-67698fc88c-t6ft7" podStartSLOduration=2.833931535 podStartE2EDuration="2.833931535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:18:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:18:10.83210496 +0000 UTC m=+3194.350420457" watchObservedRunningTime="2026-10-30 20:18:10.833931535 +0000 UTC m=+3194.352246992" Oct 30 20:18:11.583730 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:11.583696 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-0b9215-t42cm"] Oct 30 20:18:11.587652 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:11.587628 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-0b9215-t42cm" Oct 30 20:18:11.641457 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:11.641414 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:18:11.642441 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:11.642415 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-0b9215-t42cm"] Oct 30 20:18:11.667400 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:11.667362 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpxlp\" (UniqueName: \"kubernetes.io/projected/65c9cec4-6fe9-4789-9fcc-56fda1bd4f39-kube-api-access-qpxlp\") pod \"maas-api-revoke-keys-e2e-praxis-b-0b9215-t42cm\" (UID: \"65c9cec4-6fe9-4789-9fcc-56fda1bd4f39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-0b9215-t42cm" Oct 30 20:18:11.667557 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:11.667415 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65c9cec4-6fe9-4789-9fcc-56fda1bd4f39-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-0b9215-t42cm\" (UID: \"65c9cec4-6fe9-4789-9fcc-56fda1bd4f39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-0b9215-t42cm" Oct 30 20:18:11.667648 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:11.667623 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65c9cec4-6fe9-4789-9fcc-56fda1bd4f39-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-0b9215-t42cm\" (UID: \"65c9cec4-6fe9-4789-9fcc-56fda1bd4f39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-0b9215-t42cm" Oct 30 20:18:11.689756 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:11.689733 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\"" Oct 30 20:18:11.768439 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:11.768400 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpxlp\" (UniqueName: \"kubernetes.io/projected/65c9cec4-6fe9-4789-9fcc-56fda1bd4f39-kube-api-access-qpxlp\") pod \"maas-api-revoke-keys-e2e-praxis-b-0b9215-t42cm\" (UID: \"65c9cec4-6fe9-4789-9fcc-56fda1bd4f39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-0b9215-t42cm" Oct 30 20:18:11.768439 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:11.768441 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65c9cec4-6fe9-4789-9fcc-56fda1bd4f39-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-0b9215-t42cm\" (UID: \"65c9cec4-6fe9-4789-9fcc-56fda1bd4f39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-0b9215-t42cm" Oct 30 20:18:11.768850 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:11.768528 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65c9cec4-6fe9-4789-9fcc-56fda1bd4f39-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-0b9215-t42cm\" (UID: \"65c9cec4-6fe9-4789-9fcc-56fda1bd4f39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-0b9215-t42cm" Oct 30 20:18:11.769212 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:11.769189 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65c9cec4-6fe9-4789-9fcc-56fda1bd4f39-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-0b9215-t42cm\" (UID: \"65c9cec4-6fe9-4789-9fcc-56fda1bd4f39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-0b9215-t42cm" Oct 30 20:18:11.770901 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:11.770876 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65c9cec4-6fe9-4789-9fcc-56fda1bd4f39-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-0b9215-t42cm\" (UID: \"65c9cec4-6fe9-4789-9fcc-56fda1bd4f39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-0b9215-t42cm" Oct 30 20:18:11.831225 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:11.831189 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpxlp\" (UniqueName: \"kubernetes.io/projected/65c9cec4-6fe9-4789-9fcc-56fda1bd4f39-kube-api-access-qpxlp\") pod \"maas-api-revoke-keys-e2e-praxis-b-0b9215-t42cm\" (UID: \"65c9cec4-6fe9-4789-9fcc-56fda1bd4f39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-0b9215-t42cm" Oct 30 20:18:11.900456 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:11.900342 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-0b9215-t42cm" Oct 30 20:18:12.279429 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:12.279407 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-0b9215-t42cm"] Oct 30 20:18:12.280960 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:18:12.280929 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65c9cec4_6fe9_4789_9fcc_56fda1bd4f39.slice/crio-18547df42e88e48eaf6ad346087331d0903fad901aae0d6cfb7b5b28f65dda57 WatchSource:0}: Error finding container 18547df42e88e48eaf6ad346087331d0903fad901aae0d6cfb7b5b28f65dda57: Status 404 returned error can't find the container with id 18547df42e88e48eaf6ad346087331d0903fad901aae0d6cfb7b5b28f65dda57 Oct 30 20:18:12.725882 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:12.725835 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-0b9215-t42cm" event={"ID":"65c9cec4-6fe9-4789-9fcc-56fda1bd4f39","Type":"ContainerStarted","Data":"83920ddaa203d81fab042b15129afc8c7fa80670f7a103156d9b984d3123ea99"} Oct 30 20:18:12.726084 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:12.725892 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-0b9215-t42cm" event={"ID":"65c9cec4-6fe9-4789-9fcc-56fda1bd4f39","Type":"ContainerStarted","Data":"18547df42e88e48eaf6ad346087331d0903fad901aae0d6cfb7b5b28f65dda57"} Oct 30 20:18:12.864578 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:12.864527 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-0b9215-t42cm" podStartSLOduration=1.864513492 podStartE2EDuration="1.864513492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:18:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:18:12.862806695 +0000 UTC m=+3196.381122174" watchObservedRunningTime="2026-10-30 20:18:12.864513492 +0000 UTC m=+3196.382828949" Oct 30 20:18:13.252879 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:18:13.252843 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:18:13.733330 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:13.733292 4292 generic.go:356] "Generic (PLEG): container finished" podID="65c9cec4-6fe9-4789-9fcc-56fda1bd4f39" containerID="83920ddaa203d81fab042b15129afc8c7fa80670f7a103156d9b984d3123ea99" exitCode=0 Oct 30 20:18:13.733530 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:13.733383 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-0b9215-t42cm" event={"ID":"65c9cec4-6fe9-4789-9fcc-56fda1bd4f39","Type":"ContainerDied","Data":"83920ddaa203d81fab042b15129afc8c7fa80670f7a103156d9b984d3123ea99"} Oct 30 20:18:14.882842 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:14.882816 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-0b9215-t42cm" Oct 30 20:18:14.910196 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:14.910168 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65c9cec4-6fe9-4789-9fcc-56fda1bd4f39-kube-api-access-qpxlp\" (UniqueName: \"kubernetes.io/projected/65c9cec4-6fe9-4789-9fcc-56fda1bd4f39-kube-api-access-qpxlp\") pod \"65c9cec4-6fe9-4789-9fcc-56fda1bd4f39\" (UID: \"65c9cec4-6fe9-4789-9fcc-56fda1bd4f39\") " Oct 30 20:18:14.910409 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:14.910236 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65c9cec4-6fe9-4789-9fcc-56fda1bd4f39-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65c9cec4-6fe9-4789-9fcc-56fda1bd4f39-maas-api-xks-ca\") pod \"65c9cec4-6fe9-4789-9fcc-56fda1bd4f39\" (UID: \"65c9cec4-6fe9-4789-9fcc-56fda1bd4f39\") " Oct 30 20:18:14.910409 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:14.910265 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65c9cec4-6fe9-4789-9fcc-56fda1bd4f39-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65c9cec4-6fe9-4789-9fcc-56fda1bd4f39-maas-api-openshift-ca\") pod \"65c9cec4-6fe9-4789-9fcc-56fda1bd4f39\" (UID: \"65c9cec4-6fe9-4789-9fcc-56fda1bd4f39\") " Oct 30 20:18:14.910678 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:14.910655 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65c9cec4-6fe9-4789-9fcc-56fda1bd4f39-maas-api-openshift-ca" pod "65c9cec4-6fe9-4789-9fcc-56fda1bd4f39" (UID: "65c9cec4-6fe9-4789-9fcc-56fda1bd4f39"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:18:14.912336 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:14.912314 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65c9cec4-6fe9-4789-9fcc-56fda1bd4f39-maas-api-xks-ca" pod "65c9cec4-6fe9-4789-9fcc-56fda1bd4f39" (UID: "65c9cec4-6fe9-4789-9fcc-56fda1bd4f39"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:18:14.912558 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:14.912528 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65c9cec4-6fe9-4789-9fcc-56fda1bd4f39-kube-api-access-qpxlp" pod "65c9cec4-6fe9-4789-9fcc-56fda1bd4f39" (UID: "65c9cec4-6fe9-4789-9fcc-56fda1bd4f39"). InnerVolumeSpecName "kube-api-access-qpxlp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:18:15.011610 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:15.011525 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65c9cec4-6fe9-4789-9fcc-56fda1bd4f39-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:15.011610 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:15.011562 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65c9cec4-6fe9-4789-9fcc-56fda1bd4f39-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:15.011610 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:15.011580 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpxlp\" (UniqueName: \"kubernetes.io/projected/65c9cec4-6fe9-4789-9fcc-56fda1bd4f39-kube-api-access-qpxlp\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:15.747298 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:15.747259 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-0b9215-t42cm" event={"ID":"65c9cec4-6fe9-4789-9fcc-56fda1bd4f39","Type":"ContainerDied","Data":"18547df42e88e48eaf6ad346087331d0903fad901aae0d6cfb7b5b28f65dda57"} Oct 30 20:18:15.747506 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:15.747306 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18547df42e88e48eaf6ad346087331d0903fad901aae0d6cfb7b5b28f65dda57" Oct 30 20:18:15.747506 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:15.747275 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-0b9215-t42cm" Oct 30 20:18:21.722302 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:21.722274 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e4864d-67698fc88c-t6ft7" Oct 30 20:18:29.094731 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:29.094684 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a63e7c65-hl8c5"] Oct 30 20:18:29.143494 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:29.143451 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a63e7c65-hl8c5"] Oct 30 20:18:29.882241 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:29.882206 4292 generic.go:356] "Generic (PLEG): container finished" podID="ec42d060-8ef1-46d2-9bb8-4ebb4224389b" containerID="fa7e62302e8ab4f89ed8e1c71a650a09ad421e825e4239ec7e711a8dbe778d92" exitCode=0 Oct 30 20:18:29.882506 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:29.882274 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh" event={"ID":"ec42d060-8ef1-46d2-9bb8-4ebb4224389b","Type":"ContainerDied","Data":"fa7e62302e8ab4f89ed8e1c71a650a09ad421e825e4239ec7e711a8dbe778d92"} Oct 30 20:18:29.882709 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:29.882695 4292 scope.go:122] "RemoveContainer" containerID="fa7e62302e8ab4f89ed8e1c71a650a09ad421e825e4239ec7e711a8dbe778d92" Oct 30 20:18:29.884145 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:29.884118 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:18:29.884203 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:29.884184 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:18:29.884239 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:29.884214 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:18:30.585674 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:30.585640 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0587dd2-1d22-4a53-9b86-79cb6bcbe1a2" path="/var/lib/kubelet/pods/a0587dd2-1d22-4a53-9b86-79cb6bcbe1a2/volumes" Oct 30 20:18:30.890579 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:30.890490 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh" event={"ID":"ec42d060-8ef1-46d2-9bb8-4ebb4224389b","Type":"ContainerStarted","Data":"56ca070e9b9b4b4d7e16e0a030304d51db75323d387f17b6af1d515efbee4550"} Oct 30 20:18:31.174714 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:31.171760 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh" Oct 30 20:18:31.175093 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:31.175059 4292 patch_prober.go:29] interesting pod/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Oct 30 20:18:31.175229 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:31.175124 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh" podUID="ec42d060-8ef1-46d2-9bb8-4ebb4224389b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Oct 30 20:18:31.991256 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:31.991216 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh" Oct 30 20:18:32.170825 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:32.170790 4292 patch_prober.go:29] interesting pod/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Oct 30 20:18:32.171107 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:32.171082 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh" podUID="ec42d060-8ef1-46d2-9bb8-4ebb4224389b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Oct 30 20:18:33.168861 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:33.168820 4292 patch_prober.go:29] interesting pod/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Oct 30 20:18:33.169528 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:33.169491 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh" podUID="ec42d060-8ef1-46d2-9bb8-4ebb4224389b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Oct 30 20:18:34.173377 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:34.172794 4292 patch_prober.go:29] interesting pod/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Oct 30 20:18:34.173377 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:34.172874 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh" podUID="ec42d060-8ef1-46d2-9bb8-4ebb4224389b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Oct 30 20:18:35.162186 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:35.162148 4292 patch_prober.go:29] interesting pod/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Oct 30 20:18:35.162489 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:35.162458 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh" podUID="ec42d060-8ef1-46d2-9bb8-4ebb4224389b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Oct 30 20:18:36.243877 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:36.243835 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh" Oct 30 20:18:36.245366 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:36.245329 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh" Oct 30 20:18:43.220408 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:18:43.220332 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:18:46.555017 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:46.554973 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk"] Oct 30 20:18:46.555776 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:46.555756 4292 cpu_manager.go:414] "containerMap: removing container" podUID="65c9cec4-6fe9-4789-9fcc-56fda1bd4f39" containerName="revoke-keys" Oct 30 20:18:46.555824 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:46.555780 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65c9cec4-6fe9-4789-9fcc-56fda1bd4f39" containerName="revoke-keys" Oct 30 20:18:46.555897 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:46.555885 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="65c9cec4-6fe9-4789-9fcc-56fda1bd4f39" containerName="revoke-keys" Oct 30 20:18:46.561271 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:46.561249 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk" Oct 30 20:18:46.597987 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:46.593131 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f-istio-podinfo\") pod \"e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk\" (UID: \"05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f\") " pod="openshift-ingress/e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk" Oct 30 20:18:46.597987 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:46.593179 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f-istiod-ca-cert\") pod \"e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk\" (UID: \"05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f\") " pod="openshift-ingress/e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk" Oct 30 20:18:46.597987 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:46.593283 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f-workload-certs\") pod \"e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk\" (UID: \"05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f\") " pod="openshift-ingress/e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk" Oct 30 20:18:46.597987 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:46.593315 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f-istio-token\") pod \"e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk\" (UID: \"05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f\") " pod="openshift-ingress/e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk" Oct 30 20:18:46.597987 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:46.593371 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrr2q\" (UniqueName: \"kubernetes.io/projected/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f-kube-api-access-xrr2q\") pod \"e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk\" (UID: \"05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f\") " pod="openshift-ingress/e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk" Oct 30 20:18:46.597987 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:46.593438 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f-istio-data\") pod \"e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk\" (UID: \"05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f\") " pod="openshift-ingress/e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk" Oct 30 20:18:46.597987 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:46.593468 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f-credential-socket\") pod \"e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk\" (UID: \"05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f\") " pod="openshift-ingress/e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk" Oct 30 20:18:46.597987 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:46.593493 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f-istio-envoy\") pod \"e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk\" (UID: \"05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f\") " pod="openshift-ingress/e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk" Oct 30 20:18:46.597987 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:46.593533 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f-workload-socket\") pod \"e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk\" (UID: \"05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f\") " pod="openshift-ingress/e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk" Oct 30 20:18:46.716446 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:46.666822 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk"] Oct 30 20:18:46.808754 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:46.718101 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f-workload-certs\") pod \"e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk\" (UID: \"05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f\") " pod="openshift-ingress/e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk" Oct 30 20:18:46.808754 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:46.808667 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f-istio-token\") pod \"e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk\" (UID: \"05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f\") " pod="openshift-ingress/e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk" Oct 30 20:18:46.808754 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:46.808701 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrr2q\" (UniqueName: \"kubernetes.io/projected/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f-kube-api-access-xrr2q\") pod \"e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk\" (UID: \"05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f\") " pod="openshift-ingress/e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk" Oct 30 20:18:46.809047 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:46.808801 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f-istio-data\") pod \"e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk\" (UID: \"05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f\") " pod="openshift-ingress/e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk" Oct 30 20:18:46.809047 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:46.808831 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f-credential-socket\") pod \"e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk\" (UID: \"05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f\") " pod="openshift-ingress/e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk" Oct 30 20:18:46.809047 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:46.808852 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f-istio-envoy\") pod \"e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk\" (UID: \"05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f\") " pod="openshift-ingress/e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk" Oct 30 20:18:46.809047 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:46.808910 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f-workload-socket\") pod \"e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk\" (UID: \"05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f\") " pod="openshift-ingress/e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk" Oct 30 20:18:46.809242 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:46.809067 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f-istio-podinfo\") pod \"e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk\" (UID: \"05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f\") " pod="openshift-ingress/e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk" Oct 30 20:18:46.809242 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:46.809101 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f-istiod-ca-cert\") pod \"e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk\" (UID: \"05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f\") " pod="openshift-ingress/e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk" Oct 30 20:18:46.837605 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:46.810035 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f-istiod-ca-cert\") pod \"e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk\" (UID: \"05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f\") " pod="openshift-ingress/e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk" Oct 30 20:18:46.837605 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:46.813786 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f-istio-data\") pod \"e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk\" (UID: \"05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f\") " pod="openshift-ingress/e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk" Oct 30 20:18:46.837605 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:46.718548 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f-workload-certs\") pod \"e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk\" (UID: \"05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f\") " pod="openshift-ingress/e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk" Oct 30 20:18:46.837605 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:46.778030 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-64120b-openshift-default-dockercfg-cfslr\"" Oct 30 20:18:46.837605 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:46.827326 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f-credential-socket\") pod \"e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk\" (UID: \"05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f\") " pod="openshift-ingress/e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk" Oct 30 20:18:46.837605 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:46.830461 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f-workload-socket\") pod \"e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk\" (UID: \"05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f\") " pod="openshift-ingress/e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk" Oct 30 20:18:46.837605 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:46.835636 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f-istio-envoy\") pod \"e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk\" (UID: \"05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f\") " pod="openshift-ingress/e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk" Oct 30 20:18:46.837605 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:46.836408 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f-istio-podinfo\") pod \"e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk\" (UID: \"05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f\") " pod="openshift-ingress/e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk" Oct 30 20:18:46.876708 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:46.876669 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrr2q\" (UniqueName: \"kubernetes.io/projected/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f-kube-api-access-xrr2q\") pod \"e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk\" (UID: \"05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f\") " pod="openshift-ingress/e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk" Oct 30 20:18:46.876879 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:46.876758 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f-istio-token\") pod \"e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk\" (UID: \"05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f\") " pod="openshift-ingress/e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk" Oct 30 20:18:46.922838 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:46.922797 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk" Oct 30 20:18:47.077258 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:47.077167 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b01d39e0-qk2s8"] Oct 30 20:18:47.079674 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:47.079626 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b01d39e0-qk2s8"] Oct 30 20:18:47.367201 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:47.367168 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk"] Oct 30 20:18:47.368243 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:18:47.368218 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05fc41dc_e6b5_4d5a_8a1d_cd1c3242f07f.slice/crio-ec9142b42e78b41fbe1a4add63e1ff16bd376771c5477e65b0effe258264ba9c WatchSource:0}: Error finding container ec9142b42e78b41fbe1a4add63e1ff16bd376771c5477e65b0effe258264ba9c: Status 404 returned error can't find the container with id ec9142b42e78b41fbe1a4add63e1ff16bd376771c5477e65b0effe258264ba9c Oct 30 20:18:47.370937 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:47.370904 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:18:47.371049 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:47.370981 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:18:47.371049 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:47.371013 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:18:48.034466 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:48.034423 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk" event={"ID":"05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f","Type":"ContainerStarted","Data":"d2f11e436c2c3f19ddda6c0d550ba52cb78b132fad4220f471509dfe1bbb333f"} Oct 30 20:18:48.034945 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:48.034480 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk" event={"ID":"05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f","Type":"ContainerStarted","Data":"ec9142b42e78b41fbe1a4add63e1ff16bd376771c5477e65b0effe258264ba9c"} Oct 30 20:18:48.211447 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:48.211401 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk" podStartSLOduration=2.21138431 podStartE2EDuration="2.21138431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:18:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:18:48.209531409 +0000 UTC m=+3231.727846883" watchObservedRunningTime="2026-10-30 20:18:48.21138431 +0000 UTC m=+3231.729699766" Oct 30 20:18:48.584017 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:48.583988 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04d2de80-0f9a-4a90-88b8-284e001b375c" path="/var/lib/kubelet/pods/04d2de80-0f9a-4a90-88b8-284e001b375c/volumes" Oct 30 20:18:48.923947 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:48.923881 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk" Oct 30 20:18:48.929429 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:48.929402 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk" Oct 30 20:18:49.040292 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:49.040262 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk" Oct 30 20:18:49.041372 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:18:49.041339 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk" Oct 30 20:19:03.054877 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:03.054831 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ced19c95-p7v8h"] Oct 30 20:19:03.105292 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:03.105260 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ced19c95-p7v8h"] Oct 30 20:19:04.581984 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:04.581953 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="901c66b7-8ea1-41be-9c7c-535fe0e475d9" path="/var/lib/kubelet/pods/901c66b7-8ea1-41be-9c7c-535fe0e475d9/volumes" Oct 30 20:19:05.092968 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:05.092932 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21aab5d1-2ph6l"] Oct 30 20:19:05.134046 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:05.134007 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21aab5d1-2ph6l"] Oct 30 20:19:06.582467 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:06.582436 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8cedf1d-e992-4152-9f8f-2ba60cadc3be" path="/var/lib/kubelet/pods/d8cedf1d-e992-4152-9f8f-2ba60cadc3be/volumes" Oct 30 20:19:09.981118 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:09.981084 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-0b9215-t42cm"] Oct 30 20:19:10.008088 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:10.008058 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-0b9215-t42cm"] Oct 30 20:19:10.582470 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:10.582436 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65c9cec4-6fe9-4789-9fcc-56fda1bd4f39" path="/var/lib/kubelet/pods/65c9cec4-6fe9-4789-9fcc-56fda1bd4f39/volumes" Oct 30 20:19:13.222834 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:19:13.222795 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:19:15.079650 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:15.079612 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca6a94a9-vqwph"] Oct 30 20:19:15.157102 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:15.157057 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca6a94a9-vqwph"] Oct 30 20:19:16.582638 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:16.582603 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a9a2029-1c42-4e15-8c85-f3bc00ff4536" path="/var/lib/kubelet/pods/3a9a2029-1c42-4e15-8c85-f3bc00ff4536/volumes" Oct 30 20:19:19.059514 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:19.059473 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d371e629-v8hpp"] Oct 30 20:19:19.110589 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:19.110558 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d371e629-v8hpp"] Oct 30 20:19:20.582442 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:20.582411 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65bb082c-b874-4da9-9d10-8e42f6de6576" path="/var/lib/kubelet/pods/65bb082c-b874-4da9-9d10-8e42f6de6576/volumes" Oct 30 20:19:23.696031 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:23.695997 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87e7be8d-6ghql"] Oct 30 20:19:23.701310 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:23.701285 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e7be8d-6ghql" Oct 30 20:19:23.757297 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:23.757267 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87e7be8d-6ghql"] Oct 30 20:19:23.804975 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:23.804945 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:19:23.805170 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:23.805049 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\"" Oct 30 20:19:23.822212 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:23.822172 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fca60461-40c2-422d-8b41-a72d876b7227-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87e7be8d-6ghql\" (UID: \"fca60461-40c2-422d-8b41-a72d876b7227\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e7be8d-6ghql" Oct 30 20:19:23.822454 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:23.822433 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fca60461-40c2-422d-8b41-a72d876b7227-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87e7be8d-6ghql\" (UID: \"fca60461-40c2-422d-8b41-a72d876b7227\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e7be8d-6ghql" Oct 30 20:19:23.822547 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:23.822499 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv8rn\" (UniqueName: \"kubernetes.io/projected/fca60461-40c2-422d-8b41-a72d876b7227-kube-api-access-rv8rn\") pod \"maas-api-revoke-sub-87e7be8d-6ghql\" (UID: \"fca60461-40c2-422d-8b41-a72d876b7227\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e7be8d-6ghql" Oct 30 20:19:23.923820 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:23.923793 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fca60461-40c2-422d-8b41-a72d876b7227-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87e7be8d-6ghql\" (UID: \"fca60461-40c2-422d-8b41-a72d876b7227\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e7be8d-6ghql" Oct 30 20:19:23.924013 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:23.923838 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv8rn\" (UniqueName: \"kubernetes.io/projected/fca60461-40c2-422d-8b41-a72d876b7227-kube-api-access-rv8rn\") pod \"maas-api-revoke-sub-87e7be8d-6ghql\" (UID: \"fca60461-40c2-422d-8b41-a72d876b7227\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e7be8d-6ghql" Oct 30 20:19:23.924013 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:23.923908 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fca60461-40c2-422d-8b41-a72d876b7227-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87e7be8d-6ghql\" (UID: \"fca60461-40c2-422d-8b41-a72d876b7227\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e7be8d-6ghql" Oct 30 20:19:23.924642 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:23.924620 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fca60461-40c2-422d-8b41-a72d876b7227-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87e7be8d-6ghql\" (UID: \"fca60461-40c2-422d-8b41-a72d876b7227\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e7be8d-6ghql" Oct 30 20:19:23.926292 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:23.926275 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fca60461-40c2-422d-8b41-a72d876b7227-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87e7be8d-6ghql\" (UID: \"fca60461-40c2-422d-8b41-a72d876b7227\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e7be8d-6ghql" Oct 30 20:19:23.986711 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:23.986641 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv8rn\" (UniqueName: \"kubernetes.io/projected/fca60461-40c2-422d-8b41-a72d876b7227-kube-api-access-rv8rn\") pod \"maas-api-revoke-sub-87e7be8d-6ghql\" (UID: \"fca60461-40c2-422d-8b41-a72d876b7227\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e7be8d-6ghql" Oct 30 20:19:24.016158 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:24.016117 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e7be8d-6ghql" Oct 30 20:19:24.468106 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:24.468072 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87e7be8d-6ghql"] Oct 30 20:19:24.469663 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:19:24.469636 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfca60461_40c2_422d_8b41_a72d876b7227.slice/crio-a9990fcf09e9b3e430987424b85a0ad199ad3d4ed97a503e9c50a242ea381edf WatchSource:0}: Error finding container a9990fcf09e9b3e430987424b85a0ad199ad3d4ed97a503e9c50a242ea381edf: Status 404 returned error can't find the container with id a9990fcf09e9b3e430987424b85a0ad199ad3d4ed97a503e9c50a242ea381edf Oct 30 20:19:25.329790 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:25.329760 4292 generic.go:356] "Generic (PLEG): container finished" podID="fca60461-40c2-422d-8b41-a72d876b7227" containerID="d68d51e488232e361ad48b09a7e5b43be7e0c7518d4185590fed5e4192cc00f9" exitCode=0 Oct 30 20:19:25.330151 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:25.329849 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e7be8d-6ghql" event={"ID":"fca60461-40c2-422d-8b41-a72d876b7227","Type":"ContainerDied","Data":"d68d51e488232e361ad48b09a7e5b43be7e0c7518d4185590fed5e4192cc00f9"} Oct 30 20:19:25.330151 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:25.329921 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e7be8d-6ghql" event={"ID":"fca60461-40c2-422d-8b41-a72d876b7227","Type":"ContainerStarted","Data":"a9990fcf09e9b3e430987424b85a0ad199ad3d4ed97a503e9c50a242ea381edf"} Oct 30 20:19:26.070116 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:26.070086 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45fc6b7f-hz7dl"] Oct 30 20:19:26.073724 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:26.073701 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45fc6b7f-hz7dl"] Oct 30 20:19:26.494534 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:26.494508 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e7be8d-6ghql" Oct 30 20:19:26.582667 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:26.582635 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05331ed1-fa17-4d42-81a7-9626c610a70c" path="/var/lib/kubelet/pods/05331ed1-fa17-4d42-81a7-9626c610a70c/volumes" Oct 30 20:19:26.661647 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:26.661559 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fca60461-40c2-422d-8b41-a72d876b7227-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fca60461-40c2-422d-8b41-a72d876b7227-maas-api-openshift-ca\") pod \"fca60461-40c2-422d-8b41-a72d876b7227\" (UID: \"fca60461-40c2-422d-8b41-a72d876b7227\") " Oct 30 20:19:26.661835 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:26.661664 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fca60461-40c2-422d-8b41-a72d876b7227-kube-api-access-rv8rn\" (UniqueName: \"kubernetes.io/projected/fca60461-40c2-422d-8b41-a72d876b7227-kube-api-access-rv8rn\") pod \"fca60461-40c2-422d-8b41-a72d876b7227\" (UID: \"fca60461-40c2-422d-8b41-a72d876b7227\") " Oct 30 20:19:26.661835 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:26.661742 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fca60461-40c2-422d-8b41-a72d876b7227-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fca60461-40c2-422d-8b41-a72d876b7227-maas-api-xks-ca\") pod \"fca60461-40c2-422d-8b41-a72d876b7227\" (UID: \"fca60461-40c2-422d-8b41-a72d876b7227\") " Oct 30 20:19:26.661991 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:26.661962 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fca60461-40c2-422d-8b41-a72d876b7227-maas-api-openshift-ca" pod "fca60461-40c2-422d-8b41-a72d876b7227" (UID: "fca60461-40c2-422d-8b41-a72d876b7227"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:19:26.662119 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:26.662091 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fca60461-40c2-422d-8b41-a72d876b7227-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:19:26.663858 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:26.663831 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fca60461-40c2-422d-8b41-a72d876b7227-maas-api-xks-ca" pod "fca60461-40c2-422d-8b41-a72d876b7227" (UID: "fca60461-40c2-422d-8b41-a72d876b7227"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:19:26.663977 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:26.663879 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fca60461-40c2-422d-8b41-a72d876b7227-kube-api-access-rv8rn" pod "fca60461-40c2-422d-8b41-a72d876b7227" (UID: "fca60461-40c2-422d-8b41-a72d876b7227"). InnerVolumeSpecName "kube-api-access-rv8rn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:19:26.763817 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:26.763782 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rv8rn\" (UniqueName: \"kubernetes.io/projected/fca60461-40c2-422d-8b41-a72d876b7227-kube-api-access-rv8rn\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:19:26.763817 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:26.763814 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fca60461-40c2-422d-8b41-a72d876b7227-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:19:27.343346 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:27.343307 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e7be8d-6ghql" event={"ID":"fca60461-40c2-422d-8b41-a72d876b7227","Type":"ContainerDied","Data":"a9990fcf09e9b3e430987424b85a0ad199ad3d4ed97a503e9c50a242ea381edf"} Oct 30 20:19:27.343539 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:27.343383 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9990fcf09e9b3e430987424b85a0ad199ad3d4ed97a503e9c50a242ea381edf" Oct 30 20:19:27.343539 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:27.343322 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e7be8d-6ghql" Oct 30 20:19:32.051797 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:32.051761 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3584be9e-wwcnp"] Oct 30 20:19:32.102799 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:32.102764 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3584be9e-wwcnp"] Oct 30 20:19:32.582723 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:32.582694 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c408b28f-6739-418e-89f7-f693f17f7d34" path="/var/lib/kubelet/pods/c408b28f-6739-418e-89f7-f693f17f7d34/volumes" Oct 30 20:19:35.060673 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:35.060643 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5c6b996-8nhtz"] Oct 30 20:19:35.111600 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:35.111565 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5c6b996-8nhtz"] Oct 30 20:19:36.346466 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:36.346428 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm"] Oct 30 20:19:36.346971 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:36.346956 4292 cpu_manager.go:414] "containerMap: removing container" podUID="fca60461-40c2-422d-8b41-a72d876b7227" containerName="revoke-keys" Oct 30 20:19:36.347020 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:36.346974 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fca60461-40c2-422d-8b41-a72d876b7227" containerName="revoke-keys" Oct 30 20:19:36.347083 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:36.347073 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="fca60461-40c2-422d-8b41-a72d876b7227" containerName="revoke-keys" Oct 30 20:19:36.350505 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:36.350489 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm" Oct 30 20:19:36.378382 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:36.377128 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95bc24b1-3be0-49e7-85e8-f90989e4b10b-istiod-ca-cert\") pod \"e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm\" (UID: \"95bc24b1-3be0-49e7-85e8-f90989e4b10b\") " pod="openshift-ingress/e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm" Oct 30 20:19:36.378382 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:36.377226 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95bc24b1-3be0-49e7-85e8-f90989e4b10b-credential-socket\") pod \"e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm\" (UID: \"95bc24b1-3be0-49e7-85e8-f90989e4b10b\") " pod="openshift-ingress/e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm" Oct 30 20:19:36.378382 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:36.377307 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95bc24b1-3be0-49e7-85e8-f90989e4b10b-istio-podinfo\") pod \"e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm\" (UID: \"95bc24b1-3be0-49e7-85e8-f90989e4b10b\") " pod="openshift-ingress/e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm" Oct 30 20:19:36.378382 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:36.377366 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95bc24b1-3be0-49e7-85e8-f90989e4b10b-workload-socket\") pod \"e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm\" (UID: \"95bc24b1-3be0-49e7-85e8-f90989e4b10b\") " pod="openshift-ingress/e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm" Oct 30 20:19:36.378382 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:36.377418 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95bc24b1-3be0-49e7-85e8-f90989e4b10b-istio-token\") pod \"e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm\" (UID: \"95bc24b1-3be0-49e7-85e8-f90989e4b10b\") " pod="openshift-ingress/e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm" Oct 30 20:19:36.378382 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:36.377448 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95bc24b1-3be0-49e7-85e8-f90989e4b10b-workload-certs\") pod \"e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm\" (UID: \"95bc24b1-3be0-49e7-85e8-f90989e4b10b\") " pod="openshift-ingress/e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm" Oct 30 20:19:36.378382 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:36.377491 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95bc24b1-3be0-49e7-85e8-f90989e4b10b-istio-data\") pod \"e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm\" (UID: \"95bc24b1-3be0-49e7-85e8-f90989e4b10b\") " pod="openshift-ingress/e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm" Oct 30 20:19:36.378382 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:36.377563 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2jb9\" (UniqueName: \"kubernetes.io/projected/95bc24b1-3be0-49e7-85e8-f90989e4b10b-kube-api-access-x2jb9\") pod \"e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm\" (UID: \"95bc24b1-3be0-49e7-85e8-f90989e4b10b\") " pod="openshift-ingress/e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm" Oct 30 20:19:36.378382 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:36.377609 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95bc24b1-3be0-49e7-85e8-f90989e4b10b-istio-envoy\") pod \"e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm\" (UID: \"95bc24b1-3be0-49e7-85e8-f90989e4b10b\") " pod="openshift-ingress/e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm" Oct 30 20:19:36.413801 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:36.413762 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-4b0bf3-openshift-default-dockercfg-9jxnv\"" Oct 30 20:19:36.551004 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:36.550962 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95bc24b1-3be0-49e7-85e8-f90989e4b10b-istio-data\") pod \"e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm\" (UID: \"95bc24b1-3be0-49e7-85e8-f90989e4b10b\") " pod="openshift-ingress/e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm" Oct 30 20:19:36.551312 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:36.551290 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2jb9\" (UniqueName: \"kubernetes.io/projected/95bc24b1-3be0-49e7-85e8-f90989e4b10b-kube-api-access-x2jb9\") pod \"e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm\" (UID: \"95bc24b1-3be0-49e7-85e8-f90989e4b10b\") " pod="openshift-ingress/e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm" Oct 30 20:19:36.551492 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:36.551474 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95bc24b1-3be0-49e7-85e8-f90989e4b10b-istio-envoy\") pod \"e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm\" (UID: \"95bc24b1-3be0-49e7-85e8-f90989e4b10b\") " pod="openshift-ingress/e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm" Oct 30 20:19:36.552150 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:36.552125 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95bc24b1-3be0-49e7-85e8-f90989e4b10b-istiod-ca-cert\") pod \"e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm\" (UID: \"95bc24b1-3be0-49e7-85e8-f90989e4b10b\") " pod="openshift-ingress/e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm" Oct 30 20:19:36.552319 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:36.552305 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95bc24b1-3be0-49e7-85e8-f90989e4b10b-credential-socket\") pod \"e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm\" (UID: \"95bc24b1-3be0-49e7-85e8-f90989e4b10b\") " pod="openshift-ingress/e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm" Oct 30 20:19:36.552562 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:36.552546 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95bc24b1-3be0-49e7-85e8-f90989e4b10b-istio-podinfo\") pod \"e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm\" (UID: \"95bc24b1-3be0-49e7-85e8-f90989e4b10b\") " pod="openshift-ingress/e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm" Oct 30 20:19:36.552681 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:36.552667 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95bc24b1-3be0-49e7-85e8-f90989e4b10b-workload-socket\") pod \"e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm\" (UID: \"95bc24b1-3be0-49e7-85e8-f90989e4b10b\") " pod="openshift-ingress/e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm" Oct 30 20:19:36.552822 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:36.552805 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95bc24b1-3be0-49e7-85e8-f90989e4b10b-istio-token\") pod \"e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm\" (UID: \"95bc24b1-3be0-49e7-85e8-f90989e4b10b\") " pod="openshift-ingress/e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm" Oct 30 20:19:36.552922 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:36.552910 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95bc24b1-3be0-49e7-85e8-f90989e4b10b-workload-certs\") pod \"e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm\" (UID: \"95bc24b1-3be0-49e7-85e8-f90989e4b10b\") " pod="openshift-ingress/e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm" Oct 30 20:19:36.553610 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:36.553587 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95bc24b1-3be0-49e7-85e8-f90989e4b10b-workload-certs\") pod \"e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm\" (UID: \"95bc24b1-3be0-49e7-85e8-f90989e4b10b\") " pod="openshift-ingress/e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm" Oct 30 20:19:36.561378 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:36.557913 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95bc24b1-3be0-49e7-85e8-f90989e4b10b-istiod-ca-cert\") pod \"e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm\" (UID: \"95bc24b1-3be0-49e7-85e8-f90989e4b10b\") " pod="openshift-ingress/e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm" Oct 30 20:19:36.561378 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:36.559265 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95bc24b1-3be0-49e7-85e8-f90989e4b10b-credential-socket\") pod \"e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm\" (UID: \"95bc24b1-3be0-49e7-85e8-f90989e4b10b\") " pod="openshift-ingress/e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm" Oct 30 20:19:36.562387 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:36.561716 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95bc24b1-3be0-49e7-85e8-f90989e4b10b-istio-data\") pod \"e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm\" (UID: \"95bc24b1-3be0-49e7-85e8-f90989e4b10b\") " pod="openshift-ingress/e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm" Oct 30 20:19:36.569378 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:36.562631 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95bc24b1-3be0-49e7-85e8-f90989e4b10b-workload-socket\") pod \"e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm\" (UID: \"95bc24b1-3be0-49e7-85e8-f90989e4b10b\") " pod="openshift-ingress/e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm" Oct 30 20:19:36.653226 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:36.653127 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95bc24b1-3be0-49e7-85e8-f90989e4b10b-istio-podinfo\") pod \"e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm\" (UID: \"95bc24b1-3be0-49e7-85e8-f90989e4b10b\") " pod="openshift-ingress/e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm" Oct 30 20:19:36.659574 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:36.656225 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95bc24b1-3be0-49e7-85e8-f90989e4b10b-istio-envoy\") pod \"e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm\" (UID: \"95bc24b1-3be0-49e7-85e8-f90989e4b10b\") " pod="openshift-ingress/e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm" Oct 30 20:19:36.665537 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:36.663588 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89c7d9dc-a73b-418a-bb1d-37aab103b116" path="/var/lib/kubelet/pods/89c7d9dc-a73b-418a-bb1d-37aab103b116/volumes" Oct 30 20:19:36.665537 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:36.664012 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm"] Oct 30 20:19:36.712744 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:36.704809 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2jb9\" (UniqueName: \"kubernetes.io/projected/95bc24b1-3be0-49e7-85e8-f90989e4b10b-kube-api-access-x2jb9\") pod \"e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm\" (UID: \"95bc24b1-3be0-49e7-85e8-f90989e4b10b\") " pod="openshift-ingress/e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm" Oct 30 20:19:36.714420 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:36.714313 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95bc24b1-3be0-49e7-85e8-f90989e4b10b-istio-token\") pod \"e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm\" (UID: \"95bc24b1-3be0-49e7-85e8-f90989e4b10b\") " pod="openshift-ingress/e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm" Oct 30 20:19:36.966853 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:36.966767 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm" Oct 30 20:19:37.407807 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:37.407777 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm"] Oct 30 20:19:37.409758 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:19:37.409731 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95bc24b1_3be0_49e7_85e8_f90989e4b10b.slice/crio-71ba824349bd05b6ded1d0d1697b884ba64c497ef95f993705bc7cd6f4a96df0 WatchSource:0}: Error finding container 71ba824349bd05b6ded1d0d1697b884ba64c497ef95f993705bc7cd6f4a96df0: Status 404 returned error can't find the container with id 71ba824349bd05b6ded1d0d1697b884ba64c497ef95f993705bc7cd6f4a96df0 Oct 30 20:19:37.412414 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:37.412383 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:19:37.412499 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:37.412452 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:19:37.412499 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:37.412480 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:19:37.453508 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:37.453444 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm" event={"ID":"95bc24b1-3be0-49e7-85e8-f90989e4b10b","Type":"ContainerStarted","Data":"71ba824349bd05b6ded1d0d1697b884ba64c497ef95f993705bc7cd6f4a96df0"} Oct 30 20:19:38.466657 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:38.466618 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm" event={"ID":"95bc24b1-3be0-49e7-85e8-f90989e4b10b","Type":"ContainerStarted","Data":"5322434befe396b056f2096ee7d9620699cf1bd709ee1e7aedb10655cacbb465"} Oct 30 20:19:38.696399 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:38.696328 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm" podStartSLOduration=2.696311614 podStartE2EDuration="2.696311614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:38.64366069 +0000 UTC m=+3282.161976167" watchObservedRunningTime="2026-10-30 20:19:38.696311614 +0000 UTC m=+3282.214627074" Oct 30 20:19:38.967721 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:38.967691 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm" Oct 30 20:19:38.974649 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:38.974619 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm" Oct 30 20:19:39.473076 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:39.473042 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm" Oct 30 20:19:39.474158 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:39.474141 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm" Oct 30 20:19:43.055273 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:43.055188 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8e1c336-tdkjm"] Oct 30 20:19:43.059267 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:43.059240 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8e1c336-tdkjm"] Oct 30 20:19:43.221857 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:19:43.221818 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:19:44.582088 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:44.582055 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="432c0a5c-9c82-4d91-abe6-189b090f17d9" path="/var/lib/kubelet/pods/432c0a5c-9c82-4d91-abe6-189b090f17d9/volumes" Oct 30 20:19:57.063975 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:57.063929 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66699386-qwvrm"] Oct 30 20:19:57.114104 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:57.114073 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66699386-qwvrm"] Oct 30 20:19:57.511927 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:57.511849 4292 scope.go:122] "RemoveContainer" containerID="b01c1773fcc5413240d8767fe57b5d3830087f671959521ef594713dcab92a8d" Oct 30 20:19:57.520907 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:57.520889 4292 scope.go:122] "RemoveContainer" containerID="cf158f2908fbdc7271ff07c1e8b6635feb8140eabbf8b1d52641881a46389cb1" Oct 30 20:19:57.529416 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:57.529393 4292 scope.go:122] "RemoveContainer" containerID="97d9e7a15d2c091367a4569bef153a2cd9927ac7a426117679946bb400485a25" Oct 30 20:19:57.537673 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:57.537653 4292 scope.go:122] "RemoveContainer" containerID="542a219f894044b1e8a0537c579dae8457d7edc0808f083d129710da12de7fa5" Oct 30 20:19:58.582823 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:19:58.582791 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17724efe-9918-459d-ac49-789619f7da5f" path="/var/lib/kubelet/pods/17724efe-9918-459d-ac49-789619f7da5f/volumes" Oct 30 20:20:06.054106 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:06.054068 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9c6d29b-htqdr"] Oct 30 20:20:06.104702 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:06.104673 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9c6d29b-htqdr"] Oct 30 20:20:06.582877 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:06.582837 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4374031-874c-47dd-b5a0-4bbd49dde2ba" path="/var/lib/kubelet/pods/c4374031-874c-47dd-b5a0-4bbd49dde2ba/volumes" Oct 30 20:20:13.225100 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:20:13.225061 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:20:22.005101 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:22.005063 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-547082/auth-test-model-547082-kserve-7fc64f4477-8cjgx"] Oct 30 20:20:22.009241 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:22.009223 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-547082/auth-test-model-547082-kserve-7fc64f4477-8cjgx" Oct 30 20:20:22.079465 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:22.079424 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-547082/auth-test-model-547082-kserve-7fc64f4477-8cjgx"] Oct 30 20:20:22.088188 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:22.088157 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/889dfa94-11f4-425b-8773-2ad89fd14368-tls-certs\") pod \"auth-test-model-547082-kserve-7fc64f4477-8cjgx\" (UID: \"889dfa94-11f4-425b-8773-2ad89fd14368\") " pod="ai-tenant-e2e-shared-a-w0-547082/auth-test-model-547082-kserve-7fc64f4477-8cjgx" Oct 30 20:20:22.088523 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:22.088503 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/889dfa94-11f4-425b-8773-2ad89fd14368-model-cache\") pod \"auth-test-model-547082-kserve-7fc64f4477-8cjgx\" (UID: \"889dfa94-11f4-425b-8773-2ad89fd14368\") " pod="ai-tenant-e2e-shared-a-w0-547082/auth-test-model-547082-kserve-7fc64f4477-8cjgx" Oct 30 20:20:22.088652 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:22.088558 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/889dfa94-11f4-425b-8773-2ad89fd14368-tmp-dir\") pod \"auth-test-model-547082-kserve-7fc64f4477-8cjgx\" (UID: \"889dfa94-11f4-425b-8773-2ad89fd14368\") " pod="ai-tenant-e2e-shared-a-w0-547082/auth-test-model-547082-kserve-7fc64f4477-8cjgx" Oct 30 20:20:22.088721 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:22.088625 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/889dfa94-11f4-425b-8773-2ad89fd14368-home\") pod \"auth-test-model-547082-kserve-7fc64f4477-8cjgx\" (UID: \"889dfa94-11f4-425b-8773-2ad89fd14368\") " pod="ai-tenant-e2e-shared-a-w0-547082/auth-test-model-547082-kserve-7fc64f4477-8cjgx" Oct 30 20:20:22.088795 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:22.088778 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/889dfa94-11f4-425b-8773-2ad89fd14368-dshm\") pod \"auth-test-model-547082-kserve-7fc64f4477-8cjgx\" (UID: \"889dfa94-11f4-425b-8773-2ad89fd14368\") " pod="ai-tenant-e2e-shared-a-w0-547082/auth-test-model-547082-kserve-7fc64f4477-8cjgx" Oct 30 20:20:22.088901 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:22.088881 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpf6x\" (UniqueName: \"kubernetes.io/projected/889dfa94-11f4-425b-8773-2ad89fd14368-kube-api-access-jpf6x\") pod \"auth-test-model-547082-kserve-7fc64f4477-8cjgx\" (UID: \"889dfa94-11f4-425b-8773-2ad89fd14368\") " pod="ai-tenant-e2e-shared-a-w0-547082/auth-test-model-547082-kserve-7fc64f4477-8cjgx" Oct 30 20:20:22.117626 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:22.117601 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-547082\"/\"default-dockercfg-tzwxh\"" Oct 30 20:20:22.117765 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:22.117601 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-547082\"/\"kube-root-ca.crt\"" Oct 30 20:20:22.117765 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:22.117614 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-547082\"/\"auth-test-model-547082-kserve-self-signed-certs\"" Oct 30 20:20:22.117765 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:22.117676 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-547082\"/\"openshift-service-ca.crt\"" Oct 30 20:20:22.195245 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:22.195200 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/889dfa94-11f4-425b-8773-2ad89fd14368-model-cache\") pod \"auth-test-model-547082-kserve-7fc64f4477-8cjgx\" (UID: \"889dfa94-11f4-425b-8773-2ad89fd14368\") " pod="ai-tenant-e2e-shared-a-w0-547082/auth-test-model-547082-kserve-7fc64f4477-8cjgx" Oct 30 20:20:22.225371 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:22.224463 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/889dfa94-11f4-425b-8773-2ad89fd14368-tmp-dir\") pod \"auth-test-model-547082-kserve-7fc64f4477-8cjgx\" (UID: \"889dfa94-11f4-425b-8773-2ad89fd14368\") " pod="ai-tenant-e2e-shared-a-w0-547082/auth-test-model-547082-kserve-7fc64f4477-8cjgx" Oct 30 20:20:22.225371 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:22.224557 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/889dfa94-11f4-425b-8773-2ad89fd14368-home\") pod \"auth-test-model-547082-kserve-7fc64f4477-8cjgx\" (UID: \"889dfa94-11f4-425b-8773-2ad89fd14368\") " pod="ai-tenant-e2e-shared-a-w0-547082/auth-test-model-547082-kserve-7fc64f4477-8cjgx" Oct 30 20:20:22.225371 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:22.224657 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/889dfa94-11f4-425b-8773-2ad89fd14368-dshm\") pod \"auth-test-model-547082-kserve-7fc64f4477-8cjgx\" (UID: \"889dfa94-11f4-425b-8773-2ad89fd14368\") " pod="ai-tenant-e2e-shared-a-w0-547082/auth-test-model-547082-kserve-7fc64f4477-8cjgx" Oct 30 20:20:22.225371 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:22.224683 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpf6x\" (UniqueName: \"kubernetes.io/projected/889dfa94-11f4-425b-8773-2ad89fd14368-kube-api-access-jpf6x\") pod \"auth-test-model-547082-kserve-7fc64f4477-8cjgx\" (UID: \"889dfa94-11f4-425b-8773-2ad89fd14368\") " pod="ai-tenant-e2e-shared-a-w0-547082/auth-test-model-547082-kserve-7fc64f4477-8cjgx" Oct 30 20:20:22.225371 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:22.224727 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/889dfa94-11f4-425b-8773-2ad89fd14368-tls-certs\") pod \"auth-test-model-547082-kserve-7fc64f4477-8cjgx\" (UID: \"889dfa94-11f4-425b-8773-2ad89fd14368\") " pod="ai-tenant-e2e-shared-a-w0-547082/auth-test-model-547082-kserve-7fc64f4477-8cjgx" Oct 30 20:20:22.226127 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:22.226101 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/889dfa94-11f4-425b-8773-2ad89fd14368-home\") pod \"auth-test-model-547082-kserve-7fc64f4477-8cjgx\" (UID: \"889dfa94-11f4-425b-8773-2ad89fd14368\") " pod="ai-tenant-e2e-shared-a-w0-547082/auth-test-model-547082-kserve-7fc64f4477-8cjgx" Oct 30 20:20:22.226280 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:22.195920 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/889dfa94-11f4-425b-8773-2ad89fd14368-model-cache\") pod \"auth-test-model-547082-kserve-7fc64f4477-8cjgx\" (UID: \"889dfa94-11f4-425b-8773-2ad89fd14368\") " pod="ai-tenant-e2e-shared-a-w0-547082/auth-test-model-547082-kserve-7fc64f4477-8cjgx" Oct 30 20:20:22.226668 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:22.226646 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/889dfa94-11f4-425b-8773-2ad89fd14368-tmp-dir\") pod \"auth-test-model-547082-kserve-7fc64f4477-8cjgx\" (UID: \"889dfa94-11f4-425b-8773-2ad89fd14368\") " pod="ai-tenant-e2e-shared-a-w0-547082/auth-test-model-547082-kserve-7fc64f4477-8cjgx" Oct 30 20:20:22.229774 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:22.229750 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/889dfa94-11f4-425b-8773-2ad89fd14368-dshm\") pod \"auth-test-model-547082-kserve-7fc64f4477-8cjgx\" (UID: \"889dfa94-11f4-425b-8773-2ad89fd14368\") " pod="ai-tenant-e2e-shared-a-w0-547082/auth-test-model-547082-kserve-7fc64f4477-8cjgx" Oct 30 20:20:22.232226 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:22.232197 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/889dfa94-11f4-425b-8773-2ad89fd14368-tls-certs\") pod \"auth-test-model-547082-kserve-7fc64f4477-8cjgx\" (UID: \"889dfa94-11f4-425b-8773-2ad89fd14368\") " pod="ai-tenant-e2e-shared-a-w0-547082/auth-test-model-547082-kserve-7fc64f4477-8cjgx" Oct 30 20:20:22.292467 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:22.292408 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpf6x\" (UniqueName: \"kubernetes.io/projected/889dfa94-11f4-425b-8773-2ad89fd14368-kube-api-access-jpf6x\") pod \"auth-test-model-547082-kserve-7fc64f4477-8cjgx\" (UID: \"889dfa94-11f4-425b-8773-2ad89fd14368\") " pod="ai-tenant-e2e-shared-a-w0-547082/auth-test-model-547082-kserve-7fc64f4477-8cjgx" Oct 30 20:20:22.322595 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:22.322557 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-547082/auth-test-model-547082-kserve-7fc64f4477-8cjgx" Oct 30 20:20:22.757662 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:22.757636 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-547082/auth-test-model-547082-kserve-7fc64f4477-8cjgx"] Oct 30 20:20:22.758937 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:20:22.758907 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod889dfa94_11f4_425b_8773_2ad89fd14368.slice/crio-4559c2a0d411703428b936256f00c2831518726ac679591ee8b7972b374a1328 WatchSource:0}: Error finding container 4559c2a0d411703428b936256f00c2831518726ac679591ee8b7972b374a1328: Status 404 returned error can't find the container with id 4559c2a0d411703428b936256f00c2831518726ac679591ee8b7972b374a1328 Oct 30 20:20:22.770038 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:22.770011 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-547082/auth-test-model-547082-kserve-7fc64f4477-8cjgx" event={"ID":"889dfa94-11f4-425b-8773-2ad89fd14368","Type":"ContainerStarted","Data":"4559c2a0d411703428b936256f00c2831518726ac679591ee8b7972b374a1328"} Oct 30 20:20:23.811063 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:23.811021 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-547082/auth-test-model-547082-kserve-7fc64f4477-8cjgx" event={"ID":"889dfa94-11f4-425b-8773-2ad89fd14368","Type":"ContainerStarted","Data":"6a604410c18ad44007316a7122cd90d6a57c655bd25ff054597291a442b999ec"} Oct 30 20:20:23.934936 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:23.934865 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-547082/auth-test-model-547082-kserve-7fc64f4477-8cjgx" podStartSLOduration=2.9348443509999997 podStartE2EDuration="2.934844351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:20:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:20:23.932811554 +0000 UTC m=+3327.451127034" watchObservedRunningTime="2026-10-30 20:20:23.934844351 +0000 UTC m=+3327.453159809" Oct 30 20:20:24.323606 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:24.323566 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-547082/auth-test-model-547082-kserve-7fc64f4477-8cjgx" Oct 30 20:20:25.825049 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:25.825020 4292 generic.go:356] "Generic (PLEG): container finished" podID="05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f" containerID="d2f11e436c2c3f19ddda6c0d550ba52cb78b132fad4220f471509dfe1bbb333f" exitCode=0 Oct 30 20:20:25.825487 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:25.825098 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk" event={"ID":"05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f","Type":"ContainerDied","Data":"d2f11e436c2c3f19ddda6c0d550ba52cb78b132fad4220f471509dfe1bbb333f"} Oct 30 20:20:25.825682 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:25.825663 4292 scope.go:122] "RemoveContainer" containerID="d2f11e436c2c3f19ddda6c0d550ba52cb78b132fad4220f471509dfe1bbb333f" Oct 30 20:20:25.827059 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:25.827031 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:20:25.827163 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:25.827116 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:20:25.827226 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:25.827166 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:20:26.833020 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:26.832989 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk" event={"ID":"05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f","Type":"ContainerStarted","Data":"fefa5b1bed1324ee25a7037a3d40050b2cc56acdb453ca4017422905c1a59191"} Oct 30 20:20:26.923901 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:26.923871 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk" Oct 30 20:20:26.924174 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:26.924155 4292 patch_prober.go:29] interesting pod/e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 30 20:20:26.924222 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:26.924208 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk" podUID="05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 30 20:20:27.923807 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:27.923771 4292 patch_prober.go:29] interesting pod/e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 30 20:20:27.924192 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:27.923822 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk" podUID="05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 30 20:20:28.923817 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:28.923780 4292 patch_prober.go:29] interesting pod/e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 30 20:20:28.924219 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:28.923836 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk" podUID="05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 30 20:20:29.923144 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:29.923108 4292 patch_prober.go:29] interesting pod/e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 30 20:20:29.923331 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:29.923160 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk" podUID="05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 30 20:20:30.923654 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:30.923622 4292 patch_prober.go:29] interesting pod/e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 30 20:20:30.924048 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:30.923672 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk" podUID="05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 30 20:20:31.972334 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:31.972296 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk" Oct 30 20:20:31.972899 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:31.972807 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk" Oct 30 20:20:31.974171 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:31.974149 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk" Oct 30 20:20:32.323155 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:32.323124 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-547082/auth-test-model-547082-kserve-7fc64f4477-8cjgx" Oct 30 20:20:32.335711 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:32.335687 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-547082/auth-test-model-547082-kserve-7fc64f4477-8cjgx" Oct 30 20:20:32.899950 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:32.899923 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-547082/auth-test-model-547082-kserve-7fc64f4477-8cjgx" Oct 30 20:20:41.053913 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:41.053874 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89eb1db5-hhnmg"] Oct 30 20:20:41.104627 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:41.104598 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89eb1db5-hhnmg"] Oct 30 20:20:42.582415 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:42.582387 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c36eb17f-aab3-467e-b983-daf894b5226f" path="/var/lib/kubelet/pods/c36eb17f-aab3-467e-b983-daf894b5226f/volumes" Oct 30 20:20:43.220337 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:20:43.220304 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:20:57.575883 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:57.575856 4292 scope.go:122] "RemoveContainer" containerID="1b5bed8f03ef22444d2d63c7d65bc27f3f2757b83054419783f56bd9e7352156" Oct 30 20:20:57.585270 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:57.585247 4292 scope.go:122] "RemoveContainer" containerID="191aab6c03a43670954b81a54b15dc9db6040eec65405f66f4b5bb28d3543d19" Oct 30 20:20:57.594161 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:57.594140 4292 scope.go:122] "RemoveContainer" containerID="bf6543046839a2af1c0e490df5185eca3def9309b080ac91522516b19a88c7d2" Oct 30 20:20:57.603032 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:57.603012 4292 scope.go:122] "RemoveContainer" containerID="b9eaa0190d6e29bb0abce812f2c9cac9586a608cd0e39cd82cbb934dc04ccc9a" Oct 30 20:20:57.610901 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:57.610883 4292 scope.go:122] "RemoveContainer" containerID="e014890f34708b21ae4762d5e290b22d8d22e67fb2fe6c689d0595120e647fe3" Oct 30 20:20:57.619212 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:57.619195 4292 scope.go:122] "RemoveContainer" containerID="6f17ac3fa52c5b1e1ca129ff676da9f4d892dd948566c8f81798be28e59a2c09" Oct 30 20:20:57.629856 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:57.629824 4292 scope.go:122] "RemoveContainer" containerID="210f784ac02bce26bf133a8714416bbee316a11a20d58fd306d9a0616da61b5d" Oct 30 20:20:57.637712 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:57.637683 4292 scope.go:122] "RemoveContainer" containerID="216ee02f2849761a8cd78bfe00d702a451d09a409fb97edddf6f2355095c0a1e" Oct 30 20:20:57.645516 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:57.645500 4292 scope.go:122] "RemoveContainer" containerID="114e270aa59d72f17a93bd1990eb4a6f415f8826baf866143ae1bd3dad077fd1" Oct 30 20:20:57.653324 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:20:57.653306 4292 scope.go:122] "RemoveContainer" containerID="0f9459221d8021252839f2a17d8194bfbf6c17b148e908eacddf30c85d5a93f2" Oct 30 20:21:13.220312 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:21:13.220219 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:21:24.277208 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:24.277053 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-487c5f/auth-test-model-487c5f-kserve-55dfcd6c8f-26x9l"] Oct 30 20:21:24.281191 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:24.281167 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-487c5f/auth-test-model-487c5f-kserve-55dfcd6c8f-26x9l" Oct 30 20:21:24.311189 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:24.311157 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9295a589-1cb3-4d27-8a26-eb8da390b23a-dshm\") pod \"auth-test-model-487c5f-kserve-55dfcd6c8f-26x9l\" (UID: \"9295a589-1cb3-4d27-8a26-eb8da390b23a\") " pod="ai-tenant-e2e-shared-a-w3-487c5f/auth-test-model-487c5f-kserve-55dfcd6c8f-26x9l" Oct 30 20:21:24.311376 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:24.311312 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84crc\" (UniqueName: \"kubernetes.io/projected/9295a589-1cb3-4d27-8a26-eb8da390b23a-kube-api-access-84crc\") pod \"auth-test-model-487c5f-kserve-55dfcd6c8f-26x9l\" (UID: \"9295a589-1cb3-4d27-8a26-eb8da390b23a\") " pod="ai-tenant-e2e-shared-a-w3-487c5f/auth-test-model-487c5f-kserve-55dfcd6c8f-26x9l" Oct 30 20:21:24.311462 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:24.311426 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9295a589-1cb3-4d27-8a26-eb8da390b23a-model-cache\") pod \"auth-test-model-487c5f-kserve-55dfcd6c8f-26x9l\" (UID: \"9295a589-1cb3-4d27-8a26-eb8da390b23a\") " pod="ai-tenant-e2e-shared-a-w3-487c5f/auth-test-model-487c5f-kserve-55dfcd6c8f-26x9l" Oct 30 20:21:24.311535 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:24.311518 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9295a589-1cb3-4d27-8a26-eb8da390b23a-home\") pod \"auth-test-model-487c5f-kserve-55dfcd6c8f-26x9l\" (UID: \"9295a589-1cb3-4d27-8a26-eb8da390b23a\") " pod="ai-tenant-e2e-shared-a-w3-487c5f/auth-test-model-487c5f-kserve-55dfcd6c8f-26x9l" Oct 30 20:21:24.311663 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:24.311634 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9295a589-1cb3-4d27-8a26-eb8da390b23a-tmp-dir\") pod \"auth-test-model-487c5f-kserve-55dfcd6c8f-26x9l\" (UID: \"9295a589-1cb3-4d27-8a26-eb8da390b23a\") " pod="ai-tenant-e2e-shared-a-w3-487c5f/auth-test-model-487c5f-kserve-55dfcd6c8f-26x9l" Oct 30 20:21:24.311742 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:24.311694 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9295a589-1cb3-4d27-8a26-eb8da390b23a-tls-certs\") pod \"auth-test-model-487c5f-kserve-55dfcd6c8f-26x9l\" (UID: \"9295a589-1cb3-4d27-8a26-eb8da390b23a\") " pod="ai-tenant-e2e-shared-a-w3-487c5f/auth-test-model-487c5f-kserve-55dfcd6c8f-26x9l" Oct 30 20:21:24.319697 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:24.319669 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-487c5f/auth-test-model-487c5f-kserve-55dfcd6c8f-26x9l"] Oct 30 20:21:24.379635 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:24.379595 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-487c5f\"/\"auth-test-model-487c5f-kserve-self-signed-certs\"" Oct 30 20:21:24.381858 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:24.379950 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-487c5f\"/\"default-dockercfg-k5s2c\"" Oct 30 20:21:24.381858 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:24.380062 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-487c5f\"/\"openshift-service-ca.crt\"" Oct 30 20:21:24.387759 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:24.387728 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-487c5f\"/\"kube-root-ca.crt\"" Oct 30 20:21:24.419376 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:24.417677 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9295a589-1cb3-4d27-8a26-eb8da390b23a-home\") pod \"auth-test-model-487c5f-kserve-55dfcd6c8f-26x9l\" (UID: \"9295a589-1cb3-4d27-8a26-eb8da390b23a\") " pod="ai-tenant-e2e-shared-a-w3-487c5f/auth-test-model-487c5f-kserve-55dfcd6c8f-26x9l" Oct 30 20:21:24.419376 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:24.417767 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9295a589-1cb3-4d27-8a26-eb8da390b23a-tmp-dir\") pod \"auth-test-model-487c5f-kserve-55dfcd6c8f-26x9l\" (UID: \"9295a589-1cb3-4d27-8a26-eb8da390b23a\") " pod="ai-tenant-e2e-shared-a-w3-487c5f/auth-test-model-487c5f-kserve-55dfcd6c8f-26x9l" Oct 30 20:21:24.419376 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:24.417820 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9295a589-1cb3-4d27-8a26-eb8da390b23a-tls-certs\") pod \"auth-test-model-487c5f-kserve-55dfcd6c8f-26x9l\" (UID: \"9295a589-1cb3-4d27-8a26-eb8da390b23a\") " pod="ai-tenant-e2e-shared-a-w3-487c5f/auth-test-model-487c5f-kserve-55dfcd6c8f-26x9l" Oct 30 20:21:24.419376 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:24.417864 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9295a589-1cb3-4d27-8a26-eb8da390b23a-dshm\") pod \"auth-test-model-487c5f-kserve-55dfcd6c8f-26x9l\" (UID: \"9295a589-1cb3-4d27-8a26-eb8da390b23a\") " pod="ai-tenant-e2e-shared-a-w3-487c5f/auth-test-model-487c5f-kserve-55dfcd6c8f-26x9l" Oct 30 20:21:24.419376 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:24.417945 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84crc\" (UniqueName: \"kubernetes.io/projected/9295a589-1cb3-4d27-8a26-eb8da390b23a-kube-api-access-84crc\") pod \"auth-test-model-487c5f-kserve-55dfcd6c8f-26x9l\" (UID: \"9295a589-1cb3-4d27-8a26-eb8da390b23a\") " pod="ai-tenant-e2e-shared-a-w3-487c5f/auth-test-model-487c5f-kserve-55dfcd6c8f-26x9l" Oct 30 20:21:24.419376 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:24.417988 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9295a589-1cb3-4d27-8a26-eb8da390b23a-model-cache\") pod \"auth-test-model-487c5f-kserve-55dfcd6c8f-26x9l\" (UID: \"9295a589-1cb3-4d27-8a26-eb8da390b23a\") " pod="ai-tenant-e2e-shared-a-w3-487c5f/auth-test-model-487c5f-kserve-55dfcd6c8f-26x9l" Oct 30 20:21:24.419376 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:24.418955 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9295a589-1cb3-4d27-8a26-eb8da390b23a-model-cache\") pod \"auth-test-model-487c5f-kserve-55dfcd6c8f-26x9l\" (UID: \"9295a589-1cb3-4d27-8a26-eb8da390b23a\") " pod="ai-tenant-e2e-shared-a-w3-487c5f/auth-test-model-487c5f-kserve-55dfcd6c8f-26x9l" Oct 30 20:21:24.419376 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:24.419291 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9295a589-1cb3-4d27-8a26-eb8da390b23a-home\") pod \"auth-test-model-487c5f-kserve-55dfcd6c8f-26x9l\" (UID: \"9295a589-1cb3-4d27-8a26-eb8da390b23a\") " pod="ai-tenant-e2e-shared-a-w3-487c5f/auth-test-model-487c5f-kserve-55dfcd6c8f-26x9l" Oct 30 20:21:24.419951 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:24.419851 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9295a589-1cb3-4d27-8a26-eb8da390b23a-tmp-dir\") pod \"auth-test-model-487c5f-kserve-55dfcd6c8f-26x9l\" (UID: \"9295a589-1cb3-4d27-8a26-eb8da390b23a\") " pod="ai-tenant-e2e-shared-a-w3-487c5f/auth-test-model-487c5f-kserve-55dfcd6c8f-26x9l" Oct 30 20:21:24.443386 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:24.423416 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9295a589-1cb3-4d27-8a26-eb8da390b23a-dshm\") pod \"auth-test-model-487c5f-kserve-55dfcd6c8f-26x9l\" (UID: \"9295a589-1cb3-4d27-8a26-eb8da390b23a\") " pod="ai-tenant-e2e-shared-a-w3-487c5f/auth-test-model-487c5f-kserve-55dfcd6c8f-26x9l" Oct 30 20:21:24.443386 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:24.423636 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9295a589-1cb3-4d27-8a26-eb8da390b23a-tls-certs\") pod \"auth-test-model-487c5f-kserve-55dfcd6c8f-26x9l\" (UID: \"9295a589-1cb3-4d27-8a26-eb8da390b23a\") " pod="ai-tenant-e2e-shared-a-w3-487c5f/auth-test-model-487c5f-kserve-55dfcd6c8f-26x9l" Oct 30 20:21:24.521414 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:24.520556 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84crc\" (UniqueName: \"kubernetes.io/projected/9295a589-1cb3-4d27-8a26-eb8da390b23a-kube-api-access-84crc\") pod \"auth-test-model-487c5f-kserve-55dfcd6c8f-26x9l\" (UID: \"9295a589-1cb3-4d27-8a26-eb8da390b23a\") " pod="ai-tenant-e2e-shared-a-w3-487c5f/auth-test-model-487c5f-kserve-55dfcd6c8f-26x9l" Oct 30 20:21:24.593317 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:24.593244 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-487c5f/auth-test-model-487c5f-kserve-55dfcd6c8f-26x9l" Oct 30 20:21:25.039062 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:25.039033 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-487c5f/auth-test-model-487c5f-kserve-55dfcd6c8f-26x9l"] Oct 30 20:21:25.039812 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:21:25.039791 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9295a589_1cb3_4d27_8a26_eb8da390b23a.slice/crio-68f06c2c52548c496ef6173e9fae44c4140dae316b64c332c93ca3a07074c32b WatchSource:0}: Error finding container 68f06c2c52548c496ef6173e9fae44c4140dae316b64c332c93ca3a07074c32b: Status 404 returned error can't find the container with id 68f06c2c52548c496ef6173e9fae44c4140dae316b64c332c93ca3a07074c32b Oct 30 20:21:25.225945 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:25.225914 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-487c5f/auth-test-model-487c5f-kserve-55dfcd6c8f-26x9l" event={"ID":"9295a589-1cb3-4d27-8a26-eb8da390b23a","Type":"ContainerStarted","Data":"0d155e8b2332e56b2179ff77e6d935dc28879204234cc9a0be812ad47b71ca1c"} Oct 30 20:21:25.226118 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:25.225954 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-487c5f/auth-test-model-487c5f-kserve-55dfcd6c8f-26x9l" event={"ID":"9295a589-1cb3-4d27-8a26-eb8da390b23a","Type":"ContainerStarted","Data":"68f06c2c52548c496ef6173e9fae44c4140dae316b64c332c93ca3a07074c32b"} Oct 30 20:21:25.346957 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:25.346852 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-487c5f/auth-test-model-487c5f-kserve-55dfcd6c8f-26x9l" podStartSLOduration=1.346831294 podStartE2EDuration="1.346831294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:21:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:21:25.345232459 +0000 UTC m=+3388.863547963" watchObservedRunningTime="2026-10-30 20:21:25.346831294 +0000 UTC m=+3388.865146751" Oct 30 20:21:25.594497 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:25.594462 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-487c5f/auth-test-model-487c5f-kserve-55dfcd6c8f-26x9l" Oct 30 20:21:29.052919 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:29.052887 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90608da6-729zb"] Oct 30 20:21:29.103406 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:29.103371 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90608da6-729zb"] Oct 30 20:21:29.252402 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:29.252337 4292 generic.go:356] "Generic (PLEG): container finished" podID="95bc24b1-3be0-49e7-85e8-f90989e4b10b" containerID="5322434befe396b056f2096ee7d9620699cf1bd709ee1e7aedb10655cacbb465" exitCode=0 Oct 30 20:21:29.252583 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:29.252399 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm" event={"ID":"95bc24b1-3be0-49e7-85e8-f90989e4b10b","Type":"ContainerDied","Data":"5322434befe396b056f2096ee7d9620699cf1bd709ee1e7aedb10655cacbb465"} Oct 30 20:21:29.252851 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:29.252837 4292 scope.go:122] "RemoveContainer" containerID="5322434befe396b056f2096ee7d9620699cf1bd709ee1e7aedb10655cacbb465" Oct 30 20:21:29.254505 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:29.254477 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:21:29.254589 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:29.254548 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:21:29.254589 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:29.254579 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:21:30.269484 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:30.269445 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm" event={"ID":"95bc24b1-3be0-49e7-85e8-f90989e4b10b","Type":"ContainerStarted","Data":"27c4179de42f1ca7dd5df3a2c9a617775181b6430b91376358a38b379f8f7810"} Oct 30 20:21:30.591127 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:30.591044 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="448bbea1-debf-4e60-be56-98f3f51114d8" path="/var/lib/kubelet/pods/448bbea1-debf-4e60-be56-98f3f51114d8/volumes" Oct 30 20:21:30.974443 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:30.972770 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm" Oct 30 20:21:30.974443 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:30.972899 4292 patch_prober.go:29] interesting pod/e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 30 20:21:30.974443 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:30.972960 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm" podUID="95bc24b1-3be0-49e7-85e8-f90989e4b10b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 30 20:21:31.970720 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:31.970683 4292 patch_prober.go:29] interesting pod/e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 30 20:21:31.971323 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:31.971292 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm" podUID="95bc24b1-3be0-49e7-85e8-f90989e4b10b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 30 20:21:32.968758 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:32.968720 4292 patch_prober.go:29] interesting pod/e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 30 20:21:32.969036 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:32.969009 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm" podUID="95bc24b1-3be0-49e7-85e8-f90989e4b10b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 30 20:21:33.972842 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:33.972804 4292 patch_prober.go:29] interesting pod/e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 30 20:21:33.973464 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:33.973432 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm" podUID="95bc24b1-3be0-49e7-85e8-f90989e4b10b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 30 20:21:34.594414 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:34.594155 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-487c5f/auth-test-model-487c5f-kserve-55dfcd6c8f-26x9l" Oct 30 20:21:34.606213 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:34.606188 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-487c5f/auth-test-model-487c5f-kserve-55dfcd6c8f-26x9l" Oct 30 20:21:34.977435 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:34.973744 4292 patch_prober.go:29] interesting pod/e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 30 20:21:34.977435 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:34.973807 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm" podUID="95bc24b1-3be0-49e7-85e8-f90989e4b10b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 30 20:21:35.318518 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:35.318490 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-487c5f/auth-test-model-487c5f-kserve-55dfcd6c8f-26x9l" Oct 30 20:21:35.972076 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:35.972047 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm" Oct 30 20:21:35.972344 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:35.972321 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm" Oct 30 20:21:35.973009 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:35.972986 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm" Oct 30 20:21:43.220272 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:21:43.220235 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:21:51.240527 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:51.240493 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-cafbb2/auth-test-model-cafbb2-kserve-96fc794d9-g6gkh"] Oct 30 20:21:51.244227 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:51.244208 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-cafbb2/auth-test-model-cafbb2-kserve-96fc794d9-g6gkh" Oct 30 20:21:51.300595 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:51.300562 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-cafbb2/auth-test-model-cafbb2-kserve-96fc794d9-g6gkh"] Oct 30 20:21:51.347070 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:51.347040 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-cafbb2\"/\"openshift-service-ca.crt\"" Oct 30 20:21:51.347070 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:51.347057 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-cafbb2\"/\"kube-root-ca.crt\"" Oct 30 20:21:51.348511 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:51.348489 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-cafbb2\"/\"auth-test-model-cafbb2-kserve-self-signed-certs\"" Oct 30 20:21:51.348644 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:51.348540 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-cafbb2\"/\"default-dockercfg-vhjvv\"" Oct 30 20:21:51.415376 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:51.411377 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71a33844-736e-4eeb-b9b9-989bd68a6db3-model-cache\") pod \"auth-test-model-cafbb2-kserve-96fc794d9-g6gkh\" (UID: \"71a33844-736e-4eeb-b9b9-989bd68a6db3\") " pod="ai-tenant-e2e-shared-a-w1-cafbb2/auth-test-model-cafbb2-kserve-96fc794d9-g6gkh" Oct 30 20:21:51.415376 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:51.411465 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71a33844-736e-4eeb-b9b9-989bd68a6db3-tmp-dir\") pod \"auth-test-model-cafbb2-kserve-96fc794d9-g6gkh\" (UID: \"71a33844-736e-4eeb-b9b9-989bd68a6db3\") " pod="ai-tenant-e2e-shared-a-w1-cafbb2/auth-test-model-cafbb2-kserve-96fc794d9-g6gkh" Oct 30 20:21:51.415376 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:51.411504 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x7fs\" (UniqueName: \"kubernetes.io/projected/71a33844-736e-4eeb-b9b9-989bd68a6db3-kube-api-access-6x7fs\") pod \"auth-test-model-cafbb2-kserve-96fc794d9-g6gkh\" (UID: \"71a33844-736e-4eeb-b9b9-989bd68a6db3\") " pod="ai-tenant-e2e-shared-a-w1-cafbb2/auth-test-model-cafbb2-kserve-96fc794d9-g6gkh" Oct 30 20:21:51.415376 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:51.411538 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71a33844-736e-4eeb-b9b9-989bd68a6db3-dshm\") pod \"auth-test-model-cafbb2-kserve-96fc794d9-g6gkh\" (UID: \"71a33844-736e-4eeb-b9b9-989bd68a6db3\") " pod="ai-tenant-e2e-shared-a-w1-cafbb2/auth-test-model-cafbb2-kserve-96fc794d9-g6gkh" Oct 30 20:21:51.415376 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:51.411615 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71a33844-736e-4eeb-b9b9-989bd68a6db3-tls-certs\") pod \"auth-test-model-cafbb2-kserve-96fc794d9-g6gkh\" (UID: \"71a33844-736e-4eeb-b9b9-989bd68a6db3\") " pod="ai-tenant-e2e-shared-a-w1-cafbb2/auth-test-model-cafbb2-kserve-96fc794d9-g6gkh" Oct 30 20:21:51.425722 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:51.419500 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71a33844-736e-4eeb-b9b9-989bd68a6db3-home\") pod \"auth-test-model-cafbb2-kserve-96fc794d9-g6gkh\" (UID: \"71a33844-736e-4eeb-b9b9-989bd68a6db3\") " pod="ai-tenant-e2e-shared-a-w1-cafbb2/auth-test-model-cafbb2-kserve-96fc794d9-g6gkh" Oct 30 20:21:51.521020 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:51.520975 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71a33844-736e-4eeb-b9b9-989bd68a6db3-model-cache\") pod \"auth-test-model-cafbb2-kserve-96fc794d9-g6gkh\" (UID: \"71a33844-736e-4eeb-b9b9-989bd68a6db3\") " pod="ai-tenant-e2e-shared-a-w1-cafbb2/auth-test-model-cafbb2-kserve-96fc794d9-g6gkh" Oct 30 20:21:51.521217 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:51.521024 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71a33844-736e-4eeb-b9b9-989bd68a6db3-tmp-dir\") pod \"auth-test-model-cafbb2-kserve-96fc794d9-g6gkh\" (UID: \"71a33844-736e-4eeb-b9b9-989bd68a6db3\") " pod="ai-tenant-e2e-shared-a-w1-cafbb2/auth-test-model-cafbb2-kserve-96fc794d9-g6gkh" Oct 30 20:21:51.521217 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:51.521057 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x7fs\" (UniqueName: \"kubernetes.io/projected/71a33844-736e-4eeb-b9b9-989bd68a6db3-kube-api-access-6x7fs\") pod \"auth-test-model-cafbb2-kserve-96fc794d9-g6gkh\" (UID: \"71a33844-736e-4eeb-b9b9-989bd68a6db3\") " pod="ai-tenant-e2e-shared-a-w1-cafbb2/auth-test-model-cafbb2-kserve-96fc794d9-g6gkh" Oct 30 20:21:51.521217 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:51.521092 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71a33844-736e-4eeb-b9b9-989bd68a6db3-dshm\") pod \"auth-test-model-cafbb2-kserve-96fc794d9-g6gkh\" (UID: \"71a33844-736e-4eeb-b9b9-989bd68a6db3\") " pod="ai-tenant-e2e-shared-a-w1-cafbb2/auth-test-model-cafbb2-kserve-96fc794d9-g6gkh" Oct 30 20:21:51.521217 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:51.521153 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71a33844-736e-4eeb-b9b9-989bd68a6db3-tls-certs\") pod \"auth-test-model-cafbb2-kserve-96fc794d9-g6gkh\" (UID: \"71a33844-736e-4eeb-b9b9-989bd68a6db3\") " pod="ai-tenant-e2e-shared-a-w1-cafbb2/auth-test-model-cafbb2-kserve-96fc794d9-g6gkh" Oct 30 20:21:51.521217 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:51.521211 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71a33844-736e-4eeb-b9b9-989bd68a6db3-home\") pod \"auth-test-model-cafbb2-kserve-96fc794d9-g6gkh\" (UID: \"71a33844-736e-4eeb-b9b9-989bd68a6db3\") " pod="ai-tenant-e2e-shared-a-w1-cafbb2/auth-test-model-cafbb2-kserve-96fc794d9-g6gkh" Oct 30 20:21:51.521539 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:51.521458 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71a33844-736e-4eeb-b9b9-989bd68a6db3-tmp-dir\") pod \"auth-test-model-cafbb2-kserve-96fc794d9-g6gkh\" (UID: \"71a33844-736e-4eeb-b9b9-989bd68a6db3\") " pod="ai-tenant-e2e-shared-a-w1-cafbb2/auth-test-model-cafbb2-kserve-96fc794d9-g6gkh" Oct 30 20:21:51.521539 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:51.521466 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71a33844-736e-4eeb-b9b9-989bd68a6db3-model-cache\") pod \"auth-test-model-cafbb2-kserve-96fc794d9-g6gkh\" (UID: \"71a33844-736e-4eeb-b9b9-989bd68a6db3\") " pod="ai-tenant-e2e-shared-a-w1-cafbb2/auth-test-model-cafbb2-kserve-96fc794d9-g6gkh" Oct 30 20:21:51.521831 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:51.521803 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71a33844-736e-4eeb-b9b9-989bd68a6db3-home\") pod \"auth-test-model-cafbb2-kserve-96fc794d9-g6gkh\" (UID: \"71a33844-736e-4eeb-b9b9-989bd68a6db3\") " pod="ai-tenant-e2e-shared-a-w1-cafbb2/auth-test-model-cafbb2-kserve-96fc794d9-g6gkh" Oct 30 20:21:51.524058 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:51.524031 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71a33844-736e-4eeb-b9b9-989bd68a6db3-dshm\") pod \"auth-test-model-cafbb2-kserve-96fc794d9-g6gkh\" (UID: \"71a33844-736e-4eeb-b9b9-989bd68a6db3\") " pod="ai-tenant-e2e-shared-a-w1-cafbb2/auth-test-model-cafbb2-kserve-96fc794d9-g6gkh" Oct 30 20:21:51.524462 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:51.524440 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71a33844-736e-4eeb-b9b9-989bd68a6db3-tls-certs\") pod \"auth-test-model-cafbb2-kserve-96fc794d9-g6gkh\" (UID: \"71a33844-736e-4eeb-b9b9-989bd68a6db3\") " pod="ai-tenant-e2e-shared-a-w1-cafbb2/auth-test-model-cafbb2-kserve-96fc794d9-g6gkh" Oct 30 20:21:51.584295 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:51.584259 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x7fs\" (UniqueName: \"kubernetes.io/projected/71a33844-736e-4eeb-b9b9-989bd68a6db3-kube-api-access-6x7fs\") pod \"auth-test-model-cafbb2-kserve-96fc794d9-g6gkh\" (UID: \"71a33844-736e-4eeb-b9b9-989bd68a6db3\") " pod="ai-tenant-e2e-shared-a-w1-cafbb2/auth-test-model-cafbb2-kserve-96fc794d9-g6gkh" Oct 30 20:21:51.856477 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:51.856385 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-cafbb2/auth-test-model-cafbb2-kserve-96fc794d9-g6gkh" Oct 30 20:21:52.238903 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:52.238875 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-cafbb2/auth-test-model-cafbb2-kserve-96fc794d9-g6gkh"] Oct 30 20:21:52.240135 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:21:52.240105 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod71a33844_736e_4eeb_b9b9_989bd68a6db3.slice/crio-d146055405d6127e16a36f66c39dc0c34cddff55b4eb307ab6ff58019e838a4f WatchSource:0}: Error finding container d146055405d6127e16a36f66c39dc0c34cddff55b4eb307ab6ff58019e838a4f: Status 404 returned error can't find the container with id d146055405d6127e16a36f66c39dc0c34cddff55b4eb307ab6ff58019e838a4f Oct 30 20:21:52.460131 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:52.460087 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-cafbb2/auth-test-model-cafbb2-kserve-96fc794d9-g6gkh" event={"ID":"71a33844-736e-4eeb-b9b9-989bd68a6db3","Type":"ContainerStarted","Data":"7f24272ab4f605d494abd486721fcc76599bdf6b64150132134460555d5fc087"} Oct 30 20:21:52.460131 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:52.460135 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-cafbb2/auth-test-model-cafbb2-kserve-96fc794d9-g6gkh" event={"ID":"71a33844-736e-4eeb-b9b9-989bd68a6db3","Type":"ContainerStarted","Data":"d146055405d6127e16a36f66c39dc0c34cddff55b4eb307ab6ff58019e838a4f"} Oct 30 20:21:52.581891 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:52.581841 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-cafbb2/auth-test-model-cafbb2-kserve-96fc794d9-g6gkh" podStartSLOduration=1.581826766 podStartE2EDuration="1.581826766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:21:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:21:52.578779638 +0000 UTC m=+3416.097095094" watchObservedRunningTime="2026-10-30 20:21:52.581826766 +0000 UTC m=+3416.100142223" Oct 30 20:21:52.857157 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:52.857070 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-cafbb2/auth-test-model-cafbb2-kserve-96fc794d9-g6gkh" Oct 30 20:21:57.711836 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:57.711801 4292 scope.go:122] "RemoveContainer" containerID="35fa25dcfa29c7b7fc43847f3a1c4778221cde63929a198412d58bcb15d89919" Oct 30 20:21:57.720843 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:21:57.720823 4292 scope.go:122] "RemoveContainer" containerID="b97450241ed4ebcd7b191d65370b987fd11ff0b5826acb56bec88465fb625d1d" Oct 30 20:22:01.857045 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:01.857000 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-cafbb2/auth-test-model-cafbb2-kserve-96fc794d9-g6gkh" Oct 30 20:22:01.873500 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:01.873472 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-cafbb2/auth-test-model-cafbb2-kserve-96fc794d9-g6gkh" Oct 30 20:22:02.527592 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:02.527562 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-cafbb2/auth-test-model-cafbb2-kserve-96fc794d9-g6gkh" Oct 30 20:22:12.516048 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:12.516013 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-056a27/auth-test-model-056a27-kserve-bdf98c7c-k7tgb"] Oct 30 20:22:12.519997 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:12.519968 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-056a27/auth-test-model-056a27-kserve-bdf98c7c-k7tgb" Oct 30 20:22:12.584263 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:12.584233 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-056a27/auth-test-model-056a27-kserve-bdf98c7c-k7tgb"] Oct 30 20:22:12.590045 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:12.590016 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c29e722c-2f9e-4f04-acc4-86cc0d364373-home\") pod \"auth-test-model-056a27-kserve-bdf98c7c-k7tgb\" (UID: \"c29e722c-2f9e-4f04-acc4-86cc0d364373\") " pod="ai-tenant-e2e-shared-a-w2-056a27/auth-test-model-056a27-kserve-bdf98c7c-k7tgb" Oct 30 20:22:12.590209 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:12.590156 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c29e722c-2f9e-4f04-acc4-86cc0d364373-dshm\") pod \"auth-test-model-056a27-kserve-bdf98c7c-k7tgb\" (UID: \"c29e722c-2f9e-4f04-acc4-86cc0d364373\") " pod="ai-tenant-e2e-shared-a-w2-056a27/auth-test-model-056a27-kserve-bdf98c7c-k7tgb" Oct 30 20:22:12.590268 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:12.590249 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmn7r\" (UniqueName: \"kubernetes.io/projected/c29e722c-2f9e-4f04-acc4-86cc0d364373-kube-api-access-fmn7r\") pod \"auth-test-model-056a27-kserve-bdf98c7c-k7tgb\" (UID: \"c29e722c-2f9e-4f04-acc4-86cc0d364373\") " pod="ai-tenant-e2e-shared-a-w2-056a27/auth-test-model-056a27-kserve-bdf98c7c-k7tgb" Oct 30 20:22:12.590329 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:12.590314 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c29e722c-2f9e-4f04-acc4-86cc0d364373-tmp-dir\") pod \"auth-test-model-056a27-kserve-bdf98c7c-k7tgb\" (UID: \"c29e722c-2f9e-4f04-acc4-86cc0d364373\") " pod="ai-tenant-e2e-shared-a-w2-056a27/auth-test-model-056a27-kserve-bdf98c7c-k7tgb" Oct 30 20:22:12.590402 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:12.590379 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c29e722c-2f9e-4f04-acc4-86cc0d364373-model-cache\") pod \"auth-test-model-056a27-kserve-bdf98c7c-k7tgb\" (UID: \"c29e722c-2f9e-4f04-acc4-86cc0d364373\") " pod="ai-tenant-e2e-shared-a-w2-056a27/auth-test-model-056a27-kserve-bdf98c7c-k7tgb" Oct 30 20:22:12.590458 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:12.590443 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c29e722c-2f9e-4f04-acc4-86cc0d364373-tls-certs\") pod \"auth-test-model-056a27-kserve-bdf98c7c-k7tgb\" (UID: \"c29e722c-2f9e-4f04-acc4-86cc0d364373\") " pod="ai-tenant-e2e-shared-a-w2-056a27/auth-test-model-056a27-kserve-bdf98c7c-k7tgb" Oct 30 20:22:12.623180 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:12.623149 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-056a27\"/\"auth-test-model-056a27-kserve-self-signed-certs\"" Oct 30 20:22:12.624286 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:12.624266 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-056a27\"/\"openshift-service-ca.crt\"" Oct 30 20:22:12.624286 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:12.624269 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-056a27\"/\"default-dockercfg-8r9tf\"" Oct 30 20:22:12.624560 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:12.624544 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-056a27\"/\"kube-root-ca.crt\"" Oct 30 20:22:12.707378 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:12.706308 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmn7r\" (UniqueName: \"kubernetes.io/projected/c29e722c-2f9e-4f04-acc4-86cc0d364373-kube-api-access-fmn7r\") pod \"auth-test-model-056a27-kserve-bdf98c7c-k7tgb\" (UID: \"c29e722c-2f9e-4f04-acc4-86cc0d364373\") " pod="ai-tenant-e2e-shared-a-w2-056a27/auth-test-model-056a27-kserve-bdf98c7c-k7tgb" Oct 30 20:22:12.707378 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:12.706418 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c29e722c-2f9e-4f04-acc4-86cc0d364373-tmp-dir\") pod \"auth-test-model-056a27-kserve-bdf98c7c-k7tgb\" (UID: \"c29e722c-2f9e-4f04-acc4-86cc0d364373\") " pod="ai-tenant-e2e-shared-a-w2-056a27/auth-test-model-056a27-kserve-bdf98c7c-k7tgb" Oct 30 20:22:12.707378 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:12.706470 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c29e722c-2f9e-4f04-acc4-86cc0d364373-model-cache\") pod \"auth-test-model-056a27-kserve-bdf98c7c-k7tgb\" (UID: \"c29e722c-2f9e-4f04-acc4-86cc0d364373\") " pod="ai-tenant-e2e-shared-a-w2-056a27/auth-test-model-056a27-kserve-bdf98c7c-k7tgb" Oct 30 20:22:12.707378 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:12.706531 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c29e722c-2f9e-4f04-acc4-86cc0d364373-tls-certs\") pod \"auth-test-model-056a27-kserve-bdf98c7c-k7tgb\" (UID: \"c29e722c-2f9e-4f04-acc4-86cc0d364373\") " pod="ai-tenant-e2e-shared-a-w2-056a27/auth-test-model-056a27-kserve-bdf98c7c-k7tgb" Oct 30 20:22:12.707378 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:12.706609 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c29e722c-2f9e-4f04-acc4-86cc0d364373-home\") pod \"auth-test-model-056a27-kserve-bdf98c7c-k7tgb\" (UID: \"c29e722c-2f9e-4f04-acc4-86cc0d364373\") " pod="ai-tenant-e2e-shared-a-w2-056a27/auth-test-model-056a27-kserve-bdf98c7c-k7tgb" Oct 30 20:22:12.707378 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:12.706743 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c29e722c-2f9e-4f04-acc4-86cc0d364373-dshm\") pod \"auth-test-model-056a27-kserve-bdf98c7c-k7tgb\" (UID: \"c29e722c-2f9e-4f04-acc4-86cc0d364373\") " pod="ai-tenant-e2e-shared-a-w2-056a27/auth-test-model-056a27-kserve-bdf98c7c-k7tgb" Oct 30 20:22:12.726374 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:12.713771 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c29e722c-2f9e-4f04-acc4-86cc0d364373-home\") pod \"auth-test-model-056a27-kserve-bdf98c7c-k7tgb\" (UID: \"c29e722c-2f9e-4f04-acc4-86cc0d364373\") " pod="ai-tenant-e2e-shared-a-w2-056a27/auth-test-model-056a27-kserve-bdf98c7c-k7tgb" Oct 30 20:22:12.726374 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:12.720457 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c29e722c-2f9e-4f04-acc4-86cc0d364373-model-cache\") pod \"auth-test-model-056a27-kserve-bdf98c7c-k7tgb\" (UID: \"c29e722c-2f9e-4f04-acc4-86cc0d364373\") " pod="ai-tenant-e2e-shared-a-w2-056a27/auth-test-model-056a27-kserve-bdf98c7c-k7tgb" Oct 30 20:22:12.726374 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:12.723018 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c29e722c-2f9e-4f04-acc4-86cc0d364373-tmp-dir\") pod \"auth-test-model-056a27-kserve-bdf98c7c-k7tgb\" (UID: \"c29e722c-2f9e-4f04-acc4-86cc0d364373\") " pod="ai-tenant-e2e-shared-a-w2-056a27/auth-test-model-056a27-kserve-bdf98c7c-k7tgb" Oct 30 20:22:12.778370 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:12.774904 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c29e722c-2f9e-4f04-acc4-86cc0d364373-dshm\") pod \"auth-test-model-056a27-kserve-bdf98c7c-k7tgb\" (UID: \"c29e722c-2f9e-4f04-acc4-86cc0d364373\") " pod="ai-tenant-e2e-shared-a-w2-056a27/auth-test-model-056a27-kserve-bdf98c7c-k7tgb" Oct 30 20:22:12.792380 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:12.791527 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c29e722c-2f9e-4f04-acc4-86cc0d364373-tls-certs\") pod \"auth-test-model-056a27-kserve-bdf98c7c-k7tgb\" (UID: \"c29e722c-2f9e-4f04-acc4-86cc0d364373\") " pod="ai-tenant-e2e-shared-a-w2-056a27/auth-test-model-056a27-kserve-bdf98c7c-k7tgb" Oct 30 20:22:12.801104 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:12.801041 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmn7r\" (UniqueName: \"kubernetes.io/projected/c29e722c-2f9e-4f04-acc4-86cc0d364373-kube-api-access-fmn7r\") pod \"auth-test-model-056a27-kserve-bdf98c7c-k7tgb\" (UID: \"c29e722c-2f9e-4f04-acc4-86cc0d364373\") " pod="ai-tenant-e2e-shared-a-w2-056a27/auth-test-model-056a27-kserve-bdf98c7c-k7tgb" Oct 30 20:22:12.837389 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:12.832744 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-056a27/auth-test-model-056a27-kserve-bdf98c7c-k7tgb" Oct 30 20:22:13.221941 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:22:13.221839 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:22:13.257802 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:13.257776 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-056a27/auth-test-model-056a27-kserve-bdf98c7c-k7tgb"] Oct 30 20:22:13.259788 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:22:13.259758 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc29e722c_2f9e_4f04_acc4_86cc0d364373.slice/crio-cdf4c6b79c3beebe98f30d49bd2f50bff86bf084bdfb89b533aa8d26296d8f57 WatchSource:0}: Error finding container cdf4c6b79c3beebe98f30d49bd2f50bff86bf084bdfb89b533aa8d26296d8f57: Status 404 returned error can't find the container with id cdf4c6b79c3beebe98f30d49bd2f50bff86bf084bdfb89b533aa8d26296d8f57 Oct 30 20:22:13.596336 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:13.596296 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-056a27/auth-test-model-056a27-kserve-bdf98c7c-k7tgb" event={"ID":"c29e722c-2f9e-4f04-acc4-86cc0d364373","Type":"ContainerStarted","Data":"ff9e5164fd7e5c208041fe711c5e99fa6384e3393ea3ca3ca45afaf4ba19ff7f"} Oct 30 20:22:13.596336 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:13.596342 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-056a27/auth-test-model-056a27-kserve-bdf98c7c-k7tgb" event={"ID":"c29e722c-2f9e-4f04-acc4-86cc0d364373","Type":"ContainerStarted","Data":"cdf4c6b79c3beebe98f30d49bd2f50bff86bf084bdfb89b533aa8d26296d8f57"} Oct 30 20:22:13.715055 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:13.715003 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-056a27/auth-test-model-056a27-kserve-bdf98c7c-k7tgb" podStartSLOduration=1.714990271 podStartE2EDuration="1.714990271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:22:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:22:13.714985651 +0000 UTC m=+3437.233301128" watchObservedRunningTime="2026-10-30 20:22:13.714990271 +0000 UTC m=+3437.233305729" Oct 30 20:22:13.844373 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:13.840456 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-056a27/auth-test-model-056a27-kserve-bdf98c7c-k7tgb" Oct 30 20:22:19.247879 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:19.247840 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p"] Oct 30 20:22:19.248341 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:19.248044 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p" podUID="1fb07cea-63a9-4744-9ce5-fd802f091a82" containerName="istio-proxy" containerID="cri-o://9fef014fa514f89fef83336cbeb3e719def15ef94762276d678a9db4dceba41c" gracePeriod=30 Oct 30 20:22:22.833792 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:22.833756 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-056a27/auth-test-model-056a27-kserve-bdf98c7c-k7tgb" Oct 30 20:22:22.846284 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:22.846258 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-056a27/auth-test-model-056a27-kserve-bdf98c7c-k7tgb" Oct 30 20:22:23.669809 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:23.669779 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-056a27/auth-test-model-056a27-kserve-bdf98c7c-k7tgb" Oct 30 20:22:24.511232 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:24.511176 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p" Oct 30 20:22:24.584796 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:24.584773 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fb07cea-63a9-4744-9ce5-fd802f091a82-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fb07cea-63a9-4744-9ce5-fd802f091a82-workload-certs\") pod \"1fb07cea-63a9-4744-9ce5-fd802f091a82\" (UID: \"1fb07cea-63a9-4744-9ce5-fd802f091a82\") " Oct 30 20:22:24.584942 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:24.584812 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fb07cea-63a9-4744-9ce5-fd802f091a82-istio-token\" (UniqueName: \"kubernetes.io/projected/1fb07cea-63a9-4744-9ce5-fd802f091a82-istio-token\") pod \"1fb07cea-63a9-4744-9ce5-fd802f091a82\" (UID: \"1fb07cea-63a9-4744-9ce5-fd802f091a82\") " Oct 30 20:22:24.584942 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:24.584858 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fb07cea-63a9-4744-9ce5-fd802f091a82-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb07cea-63a9-4744-9ce5-fd802f091a82-workload-socket\") pod \"1fb07cea-63a9-4744-9ce5-fd802f091a82\" (UID: \"1fb07cea-63a9-4744-9ce5-fd802f091a82\") " Oct 30 20:22:24.584942 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:24.584904 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fb07cea-63a9-4744-9ce5-fd802f091a82-kube-api-access-sq2ds\" (UniqueName: \"kubernetes.io/projected/1fb07cea-63a9-4744-9ce5-fd802f091a82-kube-api-access-sq2ds\") pod \"1fb07cea-63a9-4744-9ce5-fd802f091a82\" (UID: \"1fb07cea-63a9-4744-9ce5-fd802f091a82\") " Oct 30 20:22:24.585093 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:24.584953 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fb07cea-63a9-4744-9ce5-fd802f091a82-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb07cea-63a9-4744-9ce5-fd802f091a82-credential-socket\") pod \"1fb07cea-63a9-4744-9ce5-fd802f091a82\" (UID: \"1fb07cea-63a9-4744-9ce5-fd802f091a82\") " Oct 30 20:22:24.585093 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:24.584999 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fb07cea-63a9-4744-9ce5-fd802f091a82-workload-certs" pod "1fb07cea-63a9-4744-9ce5-fd802f091a82" (UID: "1fb07cea-63a9-4744-9ce5-fd802f091a82"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:22:24.585093 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:24.585020 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fb07cea-63a9-4744-9ce5-fd802f091a82-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fb07cea-63a9-4744-9ce5-fd802f091a82-istiod-ca-cert\") pod \"1fb07cea-63a9-4744-9ce5-fd802f091a82\" (UID: \"1fb07cea-63a9-4744-9ce5-fd802f091a82\") " Oct 30 20:22:24.585270 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:24.585112 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fb07cea-63a9-4744-9ce5-fd802f091a82-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fb07cea-63a9-4744-9ce5-fd802f091a82-istio-data\") pod \"1fb07cea-63a9-4744-9ce5-fd802f091a82\" (UID: \"1fb07cea-63a9-4744-9ce5-fd802f091a82\") " Oct 30 20:22:24.585270 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:24.585158 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fb07cea-63a9-4744-9ce5-fd802f091a82-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fb07cea-63a9-4744-9ce5-fd802f091a82-istio-envoy\") pod \"1fb07cea-63a9-4744-9ce5-fd802f091a82\" (UID: \"1fb07cea-63a9-4744-9ce5-fd802f091a82\") " Oct 30 20:22:24.585270 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:24.585160 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fb07cea-63a9-4744-9ce5-fd802f091a82-workload-socket" pod "1fb07cea-63a9-4744-9ce5-fd802f091a82" (UID: "1fb07cea-63a9-4744-9ce5-fd802f091a82"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:22:24.585270 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:24.585205 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1fb07cea-63a9-4744-9ce5-fd802f091a82-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fb07cea-63a9-4744-9ce5-fd802f091a82-istio-podinfo\") pod \"1fb07cea-63a9-4744-9ce5-fd802f091a82\" (UID: \"1fb07cea-63a9-4744-9ce5-fd802f091a82\") " Oct 30 20:22:24.585528 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:24.585255 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fb07cea-63a9-4744-9ce5-fd802f091a82-credential-socket" pod "1fb07cea-63a9-4744-9ce5-fd802f091a82" (UID: "1fb07cea-63a9-4744-9ce5-fd802f091a82"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:22:24.585528 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:24.585387 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fb07cea-63a9-4744-9ce5-fd802f091a82-istio-data" pod "1fb07cea-63a9-4744-9ce5-fd802f091a82" (UID: "1fb07cea-63a9-4744-9ce5-fd802f091a82"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:22:24.585528 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:24.585436 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fb07cea-63a9-4744-9ce5-fd802f091a82-istiod-ca-cert" pod "1fb07cea-63a9-4744-9ce5-fd802f091a82" (UID: "1fb07cea-63a9-4744-9ce5-fd802f091a82"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:22:24.585693 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:24.585659 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb07cea-63a9-4744-9ce5-fd802f091a82-credential-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:24.585693 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:24.585674 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fb07cea-63a9-4744-9ce5-fd802f091a82-istiod-ca-cert\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:24.585693 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:24.585689 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fb07cea-63a9-4744-9ce5-fd802f091a82-istio-data\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:24.585852 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:24.585703 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fb07cea-63a9-4744-9ce5-fd802f091a82-workload-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:24.585852 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:24.585712 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb07cea-63a9-4744-9ce5-fd802f091a82-workload-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:24.587166 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:24.587143 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fb07cea-63a9-4744-9ce5-fd802f091a82-kube-api-access-sq2ds" pod "1fb07cea-63a9-4744-9ce5-fd802f091a82" (UID: "1fb07cea-63a9-4744-9ce5-fd802f091a82"). InnerVolumeSpecName "kube-api-access-sq2ds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:22:24.587282 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:24.587177 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fb07cea-63a9-4744-9ce5-fd802f091a82-istio-token" pod "1fb07cea-63a9-4744-9ce5-fd802f091a82" (UID: "1fb07cea-63a9-4744-9ce5-fd802f091a82"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:22:24.587664 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:24.587642 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fb07cea-63a9-4744-9ce5-fd802f091a82-istio-envoy" pod "1fb07cea-63a9-4744-9ce5-fd802f091a82" (UID: "1fb07cea-63a9-4744-9ce5-fd802f091a82"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:22:24.587727 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:24.587669 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1fb07cea-63a9-4744-9ce5-fd802f091a82-istio-podinfo" pod "1fb07cea-63a9-4744-9ce5-fd802f091a82" (UID: "1fb07cea-63a9-4744-9ce5-fd802f091a82"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:22:24.662489 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:24.662453 4292 generic.go:356] "Generic (PLEG): container finished" podID="1fb07cea-63a9-4744-9ce5-fd802f091a82" containerID="9fef014fa514f89fef83336cbeb3e719def15ef94762276d678a9db4dceba41c" exitCode=0 Oct 30 20:22:24.662659 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:24.662512 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p" Oct 30 20:22:24.662659 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:24.662539 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p" event={"ID":"1fb07cea-63a9-4744-9ce5-fd802f091a82","Type":"ContainerDied","Data":"9fef014fa514f89fef83336cbeb3e719def15ef94762276d678a9db4dceba41c"} Oct 30 20:22:24.662659 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:24.662603 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p" event={"ID":"1fb07cea-63a9-4744-9ce5-fd802f091a82","Type":"ContainerDied","Data":"72508a584494b7eb340dae8f96f515084feb4b0bab3ea9f60c16510fa10c1bdd"} Oct 30 20:22:24.662659 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:24.662632 4292 scope.go:122] "RemoveContainer" containerID="9fef014fa514f89fef83336cbeb3e719def15ef94762276d678a9db4dceba41c" Oct 30 20:22:24.672975 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:24.672824 4292 scope.go:122] "RemoveContainer" containerID="9fef014fa514f89fef83336cbeb3e719def15ef94762276d678a9db4dceba41c" Oct 30 20:22:24.673176 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:22:24.673155 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fef014fa514f89fef83336cbeb3e719def15ef94762276d678a9db4dceba41c\": container with ID starting with 9fef014fa514f89fef83336cbeb3e719def15ef94762276d678a9db4dceba41c not found: ID does not exist" containerID="9fef014fa514f89fef83336cbeb3e719def15ef94762276d678a9db4dceba41c" Oct 30 20:22:24.673262 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:24.673188 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fef014fa514f89fef83336cbeb3e719def15ef94762276d678a9db4dceba41c"} err="failed to get container status \"9fef014fa514f89fef83336cbeb3e719def15ef94762276d678a9db4dceba41c\": rpc error: code = NotFound desc = could not find container \"9fef014fa514f89fef83336cbeb3e719def15ef94762276d678a9db4dceba41c\": container with ID starting with 9fef014fa514f89fef83336cbeb3e719def15ef94762276d678a9db4dceba41c not found: ID does not exist" Oct 30 20:22:24.686646 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:24.686608 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fb07cea-63a9-4744-9ce5-fd802f091a82-istio-envoy\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:24.686646 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:24.686634 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fb07cea-63a9-4744-9ce5-fd802f091a82-istio-podinfo\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:24.686817 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:24.686649 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fb07cea-63a9-4744-9ce5-fd802f091a82-istio-token\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:24.686817 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:24.686663 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq2ds\" (UniqueName: \"kubernetes.io/projected/1fb07cea-63a9-4744-9ce5-fd802f091a82-kube-api-access-sq2ds\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:24.807124 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:24.807093 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p"] Oct 30 20:22:24.809639 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:24.809618 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-8cj4p"] Oct 30 20:22:26.585065 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:26.585023 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fb07cea-63a9-4744-9ce5-fd802f091a82" path="/var/lib/kubelet/pods/1fb07cea-63a9-4744-9ce5-fd802f091a82/volumes" Oct 30 20:22:31.564837 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:31.564806 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr"] Oct 30 20:22:31.565293 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:31.565066 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr" podUID="623dc6cb-9a97-48df-9388-cf847d18a146" containerName="istio-proxy" containerID="cri-o://6d199db1668d4b9305b9fa05db8bc858682972d13cf53698d367c4f4ac7fffd3" gracePeriod=30 Oct 30 20:22:34.372872 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:34.372837 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb"] Oct 30 20:22:34.373519 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:34.373093 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb" podUID="494c9125-ea0a-4656-af2b-f0d198271ef3" containerName="istio-proxy" containerID="cri-o://a1fb257a32adaa69638f0a9ce7cefd59ea1be8bc6c0594bbd7725b9f71149fcb" gracePeriod=30 Oct 30 20:22:36.777572 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:36.777531 4292 patch_prober.go:29] interesting pod/e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Oct 30 20:22:36.778064 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:36.777618 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr" podUID="623dc6cb-9a97-48df-9388-cf847d18a146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Oct 30 20:22:36.800796 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:36.800756 4292 generic.go:356] "Generic (PLEG): container finished" podID="623dc6cb-9a97-48df-9388-cf847d18a146" containerID="6d199db1668d4b9305b9fa05db8bc858682972d13cf53698d367c4f4ac7fffd3" exitCode=0 Oct 30 20:22:36.800998 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:36.800826 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr" event={"ID":"623dc6cb-9a97-48df-9388-cf847d18a146","Type":"ContainerDied","Data":"6d199db1668d4b9305b9fa05db8bc858682972d13cf53698d367c4f4ac7fffd3"} Oct 30 20:22:36.918524 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:36.918500 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr" Oct 30 20:22:37.053587 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:37.053554 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/623dc6cb-9a97-48df-9388-cf847d18a146-istio-token\" (UniqueName: \"kubernetes.io/projected/623dc6cb-9a97-48df-9388-cf847d18a146-istio-token\") pod \"623dc6cb-9a97-48df-9388-cf847d18a146\" (UID: \"623dc6cb-9a97-48df-9388-cf847d18a146\") " Oct 30 20:22:37.053785 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:37.053670 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/623dc6cb-9a97-48df-9388-cf847d18a146-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/623dc6cb-9a97-48df-9388-cf847d18a146-istio-data\") pod \"623dc6cb-9a97-48df-9388-cf847d18a146\" (UID: \"623dc6cb-9a97-48df-9388-cf847d18a146\") " Oct 30 20:22:37.053785 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:37.053770 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/623dc6cb-9a97-48df-9388-cf847d18a146-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/623dc6cb-9a97-48df-9388-cf847d18a146-credential-socket\") pod \"623dc6cb-9a97-48df-9388-cf847d18a146\" (UID: \"623dc6cb-9a97-48df-9388-cf847d18a146\") " Oct 30 20:22:37.053916 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:37.053852 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/623dc6cb-9a97-48df-9388-cf847d18a146-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/623dc6cb-9a97-48df-9388-cf847d18a146-istiod-ca-cert\") pod \"623dc6cb-9a97-48df-9388-cf847d18a146\" (UID: \"623dc6cb-9a97-48df-9388-cf847d18a146\") " Oct 30 20:22:37.053916 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:37.053876 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/623dc6cb-9a97-48df-9388-cf847d18a146-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/623dc6cb-9a97-48df-9388-cf847d18a146-workload-certs\") pod \"623dc6cb-9a97-48df-9388-cf847d18a146\" (UID: \"623dc6cb-9a97-48df-9388-cf847d18a146\") " Oct 30 20:22:37.054029 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:37.053921 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/623dc6cb-9a97-48df-9388-cf847d18a146-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/623dc6cb-9a97-48df-9388-cf847d18a146-istio-podinfo\") pod \"623dc6cb-9a97-48df-9388-cf847d18a146\" (UID: \"623dc6cb-9a97-48df-9388-cf847d18a146\") " Oct 30 20:22:37.054029 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:37.053985 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/623dc6cb-9a97-48df-9388-cf847d18a146-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/623dc6cb-9a97-48df-9388-cf847d18a146-istio-envoy\") pod \"623dc6cb-9a97-48df-9388-cf847d18a146\" (UID: \"623dc6cb-9a97-48df-9388-cf847d18a146\") " Oct 30 20:22:37.054139 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:37.054018 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/623dc6cb-9a97-48df-9388-cf847d18a146-istio-data" pod "623dc6cb-9a97-48df-9388-cf847d18a146" (UID: "623dc6cb-9a97-48df-9388-cf847d18a146"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:22:37.054139 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:37.054028 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/623dc6cb-9a97-48df-9388-cf847d18a146-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/623dc6cb-9a97-48df-9388-cf847d18a146-workload-socket\") pod \"623dc6cb-9a97-48df-9388-cf847d18a146\" (UID: \"623dc6cb-9a97-48df-9388-cf847d18a146\") " Oct 30 20:22:37.054139 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:37.054104 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/623dc6cb-9a97-48df-9388-cf847d18a146-kube-api-access-g2dvc\" (UniqueName: \"kubernetes.io/projected/623dc6cb-9a97-48df-9388-cf847d18a146-kube-api-access-g2dvc\") pod \"623dc6cb-9a97-48df-9388-cf847d18a146\" (UID: \"623dc6cb-9a97-48df-9388-cf847d18a146\") " Oct 30 20:22:37.054303 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:37.054279 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/623dc6cb-9a97-48df-9388-cf847d18a146-workload-socket" pod "623dc6cb-9a97-48df-9388-cf847d18a146" (UID: "623dc6cb-9a97-48df-9388-cf847d18a146"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:22:37.054303 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:37.054288 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/623dc6cb-9a97-48df-9388-cf847d18a146-istiod-ca-cert" pod "623dc6cb-9a97-48df-9388-cf847d18a146" (UID: "623dc6cb-9a97-48df-9388-cf847d18a146"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:22:37.054438 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:37.054294 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/623dc6cb-9a97-48df-9388-cf847d18a146-workload-certs" pod "623dc6cb-9a97-48df-9388-cf847d18a146" (UID: "623dc6cb-9a97-48df-9388-cf847d18a146"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:22:37.054609 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:37.054586 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/623dc6cb-9a97-48df-9388-cf847d18a146-credential-socket" pod "623dc6cb-9a97-48df-9388-cf847d18a146" (UID: "623dc6cb-9a97-48df-9388-cf847d18a146"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:22:37.054793 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:37.054774 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/623dc6cb-9a97-48df-9388-cf847d18a146-istio-data\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:37.054914 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:37.054799 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/623dc6cb-9a97-48df-9388-cf847d18a146-credential-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:37.054914 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:37.054816 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/623dc6cb-9a97-48df-9388-cf847d18a146-istiod-ca-cert\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:37.054914 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:37.054831 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/623dc6cb-9a97-48df-9388-cf847d18a146-workload-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:37.054914 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:37.054847 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/623dc6cb-9a97-48df-9388-cf847d18a146-workload-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:37.056063 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:37.056034 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/623dc6cb-9a97-48df-9388-cf847d18a146-istio-token" pod "623dc6cb-9a97-48df-9388-cf847d18a146" (UID: "623dc6cb-9a97-48df-9388-cf847d18a146"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:22:37.056584 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:37.056560 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/623dc6cb-9a97-48df-9388-cf847d18a146-istio-envoy" pod "623dc6cb-9a97-48df-9388-cf847d18a146" (UID: "623dc6cb-9a97-48df-9388-cf847d18a146"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:22:37.056690 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:37.056633 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/623dc6cb-9a97-48df-9388-cf847d18a146-istio-podinfo" pod "623dc6cb-9a97-48df-9388-cf847d18a146" (UID: "623dc6cb-9a97-48df-9388-cf847d18a146"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:22:37.056791 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:37.056773 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/623dc6cb-9a97-48df-9388-cf847d18a146-kube-api-access-g2dvc" pod "623dc6cb-9a97-48df-9388-cf847d18a146" (UID: "623dc6cb-9a97-48df-9388-cf847d18a146"). InnerVolumeSpecName "kube-api-access-g2dvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:22:37.156402 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:37.156284 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/623dc6cb-9a97-48df-9388-cf847d18a146-istio-podinfo\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:37.156402 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:37.156317 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/623dc6cb-9a97-48df-9388-cf847d18a146-istio-envoy\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:37.156402 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:37.156331 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2dvc\" (UniqueName: \"kubernetes.io/projected/623dc6cb-9a97-48df-9388-cf847d18a146-kube-api-access-g2dvc\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:37.156402 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:37.156345 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/623dc6cb-9a97-48df-9388-cf847d18a146-istio-token\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:37.808963 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:37.808921 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr" event={"ID":"623dc6cb-9a97-48df-9388-cf847d18a146","Type":"ContainerDied","Data":"a79dc821a5b3dde6f08d5b6a73236af8f4afa784601fa801aa92298ff63ecf08"} Oct 30 20:22:37.809455 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:37.808968 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr" Oct 30 20:22:37.809455 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:37.809015 4292 scope.go:122] "RemoveContainer" containerID="6d199db1668d4b9305b9fa05db8bc858682972d13cf53698d367c4f4ac7fffd3" Oct 30 20:22:38.029120 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:38.029089 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr"] Oct 30 20:22:38.079867 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:38.079780 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-9mvqr"] Oct 30 20:22:38.582524 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:38.582494 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="623dc6cb-9a97-48df-9388-cf847d18a146" path="/var/lib/kubelet/pods/623dc6cb-9a97-48df-9388-cf847d18a146/volumes" Oct 30 20:22:39.627427 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:39.627403 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb" Oct 30 20:22:39.788822 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:39.788794 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/494c9125-ea0a-4656-af2b-f0d198271ef3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/494c9125-ea0a-4656-af2b-f0d198271ef3-credential-socket\") pod \"494c9125-ea0a-4656-af2b-f0d198271ef3\" (UID: \"494c9125-ea0a-4656-af2b-f0d198271ef3\") " Oct 30 20:22:39.789014 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:39.788857 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/494c9125-ea0a-4656-af2b-f0d198271ef3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/494c9125-ea0a-4656-af2b-f0d198271ef3-istiod-ca-cert\") pod \"494c9125-ea0a-4656-af2b-f0d198271ef3\" (UID: \"494c9125-ea0a-4656-af2b-f0d198271ef3\") " Oct 30 20:22:39.789014 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:39.788874 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/494c9125-ea0a-4656-af2b-f0d198271ef3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/494c9125-ea0a-4656-af2b-f0d198271ef3-istio-data\") pod \"494c9125-ea0a-4656-af2b-f0d198271ef3\" (UID: \"494c9125-ea0a-4656-af2b-f0d198271ef3\") " Oct 30 20:22:39.789014 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:39.788950 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/494c9125-ea0a-4656-af2b-f0d198271ef3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/494c9125-ea0a-4656-af2b-f0d198271ef3-workload-certs\") pod \"494c9125-ea0a-4656-af2b-f0d198271ef3\" (UID: \"494c9125-ea0a-4656-af2b-f0d198271ef3\") " Oct 30 20:22:39.789175 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:39.789016 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/494c9125-ea0a-4656-af2b-f0d198271ef3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/494c9125-ea0a-4656-af2b-f0d198271ef3-workload-socket\") pod \"494c9125-ea0a-4656-af2b-f0d198271ef3\" (UID: \"494c9125-ea0a-4656-af2b-f0d198271ef3\") " Oct 30 20:22:39.789175 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:39.789068 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/494c9125-ea0a-4656-af2b-f0d198271ef3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/494c9125-ea0a-4656-af2b-f0d198271ef3-istio-envoy\") pod \"494c9125-ea0a-4656-af2b-f0d198271ef3\" (UID: \"494c9125-ea0a-4656-af2b-f0d198271ef3\") " Oct 30 20:22:39.789175 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:39.789104 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/494c9125-ea0a-4656-af2b-f0d198271ef3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/494c9125-ea0a-4656-af2b-f0d198271ef3-istio-podinfo\") pod \"494c9125-ea0a-4656-af2b-f0d198271ef3\" (UID: \"494c9125-ea0a-4656-af2b-f0d198271ef3\") " Oct 30 20:22:39.789175 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:39.789117 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/494c9125-ea0a-4656-af2b-f0d198271ef3-credential-socket" pod "494c9125-ea0a-4656-af2b-f0d198271ef3" (UID: "494c9125-ea0a-4656-af2b-f0d198271ef3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:22:39.789175 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:39.789142 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/494c9125-ea0a-4656-af2b-f0d198271ef3-istio-token\" (UniqueName: \"kubernetes.io/projected/494c9125-ea0a-4656-af2b-f0d198271ef3-istio-token\") pod \"494c9125-ea0a-4656-af2b-f0d198271ef3\" (UID: \"494c9125-ea0a-4656-af2b-f0d198271ef3\") " Oct 30 20:22:39.789478 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:39.789239 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/494c9125-ea0a-4656-af2b-f0d198271ef3-kube-api-access-g727q\" (UniqueName: \"kubernetes.io/projected/494c9125-ea0a-4656-af2b-f0d198271ef3-kube-api-access-g727q\") pod \"494c9125-ea0a-4656-af2b-f0d198271ef3\" (UID: \"494c9125-ea0a-4656-af2b-f0d198271ef3\") " Oct 30 20:22:39.789478 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:39.789245 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/494c9125-ea0a-4656-af2b-f0d198271ef3-workload-certs" pod "494c9125-ea0a-4656-af2b-f0d198271ef3" (UID: "494c9125-ea0a-4656-af2b-f0d198271ef3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:22:39.789478 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:39.789283 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/494c9125-ea0a-4656-af2b-f0d198271ef3-istio-data" pod "494c9125-ea0a-4656-af2b-f0d198271ef3" (UID: "494c9125-ea0a-4656-af2b-f0d198271ef3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:22:39.789478 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:39.789397 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/494c9125-ea0a-4656-af2b-f0d198271ef3-workload-socket" pod "494c9125-ea0a-4656-af2b-f0d198271ef3" (UID: "494c9125-ea0a-4656-af2b-f0d198271ef3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:22:39.789754 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:39.789731 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/494c9125-ea0a-4656-af2b-f0d198271ef3-istiod-ca-cert" pod "494c9125-ea0a-4656-af2b-f0d198271ef3" (UID: "494c9125-ea0a-4656-af2b-f0d198271ef3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:22:39.789840 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:39.789821 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/494c9125-ea0a-4656-af2b-f0d198271ef3-credential-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:39.789907 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:39.789845 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/494c9125-ea0a-4656-af2b-f0d198271ef3-istio-data\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:39.789907 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:39.789861 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/494c9125-ea0a-4656-af2b-f0d198271ef3-workload-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:39.789907 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:39.789874 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/494c9125-ea0a-4656-af2b-f0d198271ef3-workload-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:39.791315 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:39.791290 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/494c9125-ea0a-4656-af2b-f0d198271ef3-istio-podinfo" pod "494c9125-ea0a-4656-af2b-f0d198271ef3" (UID: "494c9125-ea0a-4656-af2b-f0d198271ef3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:22:39.791578 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:39.791550 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/494c9125-ea0a-4656-af2b-f0d198271ef3-kube-api-access-g727q" pod "494c9125-ea0a-4656-af2b-f0d198271ef3" (UID: "494c9125-ea0a-4656-af2b-f0d198271ef3"). InnerVolumeSpecName "kube-api-access-g727q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:22:39.791744 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:39.791713 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/494c9125-ea0a-4656-af2b-f0d198271ef3-istio-envoy" pod "494c9125-ea0a-4656-af2b-f0d198271ef3" (UID: "494c9125-ea0a-4656-af2b-f0d198271ef3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:22:39.791790 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:39.791727 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/494c9125-ea0a-4656-af2b-f0d198271ef3-istio-token" pod "494c9125-ea0a-4656-af2b-f0d198271ef3" (UID: "494c9125-ea0a-4656-af2b-f0d198271ef3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:22:39.823529 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:39.823493 4292 generic.go:356] "Generic (PLEG): container finished" podID="494c9125-ea0a-4656-af2b-f0d198271ef3" containerID="a1fb257a32adaa69638f0a9ce7cefd59ea1be8bc6c0594bbd7725b9f71149fcb" exitCode=0 Oct 30 20:22:39.823674 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:39.823542 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb" Oct 30 20:22:39.823674 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:39.823573 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb" event={"ID":"494c9125-ea0a-4656-af2b-f0d198271ef3","Type":"ContainerDied","Data":"a1fb257a32adaa69638f0a9ce7cefd59ea1be8bc6c0594bbd7725b9f71149fcb"} Oct 30 20:22:39.823674 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:39.823623 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb" event={"ID":"494c9125-ea0a-4656-af2b-f0d198271ef3","Type":"ContainerDied","Data":"c54d7af12437dd9cb95f03b13f26e7eb864032ab40bc84803d093b25f970813f"} Oct 30 20:22:39.823674 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:39.823646 4292 scope.go:122] "RemoveContainer" containerID="a1fb257a32adaa69638f0a9ce7cefd59ea1be8bc6c0594bbd7725b9f71149fcb" Oct 30 20:22:39.833677 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:39.833658 4292 scope.go:122] "RemoveContainer" containerID="a1fb257a32adaa69638f0a9ce7cefd59ea1be8bc6c0594bbd7725b9f71149fcb" Oct 30 20:22:39.833941 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:22:39.833923 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1fb257a32adaa69638f0a9ce7cefd59ea1be8bc6c0594bbd7725b9f71149fcb\": container with ID starting with a1fb257a32adaa69638f0a9ce7cefd59ea1be8bc6c0594bbd7725b9f71149fcb not found: ID does not exist" containerID="a1fb257a32adaa69638f0a9ce7cefd59ea1be8bc6c0594bbd7725b9f71149fcb" Oct 30 20:22:39.834033 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:39.833948 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1fb257a32adaa69638f0a9ce7cefd59ea1be8bc6c0594bbd7725b9f71149fcb"} err="failed to get container status \"a1fb257a32adaa69638f0a9ce7cefd59ea1be8bc6c0594bbd7725b9f71149fcb\": rpc error: code = NotFound desc = could not find container \"a1fb257a32adaa69638f0a9ce7cefd59ea1be8bc6c0594bbd7725b9f71149fcb\": container with ID starting with a1fb257a32adaa69638f0a9ce7cefd59ea1be8bc6c0594bbd7725b9f71149fcb not found: ID does not exist" Oct 30 20:22:39.890635 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:39.890609 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/494c9125-ea0a-4656-af2b-f0d198271ef3-istio-envoy\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:39.890635 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:39.890632 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/494c9125-ea0a-4656-af2b-f0d198271ef3-istio-podinfo\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:39.890635 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:39.890643 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/494c9125-ea0a-4656-af2b-f0d198271ef3-istio-token\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:39.890913 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:39.890653 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g727q\" (UniqueName: \"kubernetes.io/projected/494c9125-ea0a-4656-af2b-f0d198271ef3-kube-api-access-g727q\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:39.890913 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:39.890663 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/494c9125-ea0a-4656-af2b-f0d198271ef3-istiod-ca-cert\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:39.997523 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:39.997491 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb"] Oct 30 20:22:40.001966 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:40.001934 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-b4pqb"] Oct 30 20:22:40.582425 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:40.582389 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="494c9125-ea0a-4656-af2b-f0d198271ef3" path="/var/lib/kubelet/pods/494c9125-ea0a-4656-af2b-f0d198271ef3/volumes" Oct 30 20:22:43.219835 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:22:43.219796 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:22:57.746324 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:57.746282 4292 scope.go:122] "RemoveContainer" containerID="8bff9be8e6638bbd1af27154c828615d8a3b0ee15ea9d0bc5369f1aa4fea5687" Oct 30 20:22:57.755502 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:22:57.755478 4292 scope.go:122] "RemoveContainer" containerID="3a3fd19a9d82513c0163be0949e58010e6ce6a481721e1b11970092b42dda768" Oct 30 20:23:13.224863 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:23:13.224824 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:23:33.963455 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:23:33.963410 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4b0bf3-mjmdc"] Oct 30 20:23:33.964126 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:23:33.964105 4292 cpu_manager.go:414] "containerMap: removing container" podUID="494c9125-ea0a-4656-af2b-f0d198271ef3" containerName="istio-proxy" Oct 30 20:23:33.964178 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:23:33.964130 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="494c9125-ea0a-4656-af2b-f0d198271ef3" containerName="istio-proxy" Oct 30 20:23:33.964178 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:23:33.964171 4292 cpu_manager.go:414] "containerMap: removing container" podUID="1fb07cea-63a9-4744-9ce5-fd802f091a82" containerName="istio-proxy" Oct 30 20:23:33.964252 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:23:33.964178 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fb07cea-63a9-4744-9ce5-fd802f091a82" containerName="istio-proxy" Oct 30 20:23:33.964252 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:23:33.964207 4292 cpu_manager.go:414] "containerMap: removing container" podUID="623dc6cb-9a97-48df-9388-cf847d18a146" containerName="istio-proxy" Oct 30 20:23:33.964252 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:23:33.964217 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="623dc6cb-9a97-48df-9388-cf847d18a146" containerName="istio-proxy" Oct 30 20:23:33.964344 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:23:33.964316 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="623dc6cb-9a97-48df-9388-cf847d18a146" containerName="istio-proxy" Oct 30 20:23:33.964344 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:23:33.964329 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="494c9125-ea0a-4656-af2b-f0d198271ef3" containerName="istio-proxy" Oct 30 20:23:33.964435 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:23:33.964342 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fb07cea-63a9-4744-9ce5-fd802f091a82" containerName="istio-proxy" Oct 30 20:23:33.968050 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:23:33.968030 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4b0bf3-mjmdc" Oct 30 20:23:34.020234 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:23:34.020203 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4b0bf3-mjmdc"] Oct 30 20:23:34.044076 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:23:34.044040 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f20d10be-ea2a-48ec-a877-64ffb303f56d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-4b0bf3-mjmdc\" (UID: \"f20d10be-ea2a-48ec-a877-64ffb303f56d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4b0bf3-mjmdc" Oct 30 20:23:34.044268 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:23:34.044174 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f20d10be-ea2a-48ec-a877-64ffb303f56d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-4b0bf3-mjmdc\" (UID: \"f20d10be-ea2a-48ec-a877-64ffb303f56d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4b0bf3-mjmdc" Oct 30 20:23:34.044378 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:23:34.044285 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6lw7\" (UniqueName: \"kubernetes.io/projected/f20d10be-ea2a-48ec-a877-64ffb303f56d-kube-api-access-q6lw7\") pod \"maas-api-revoke-keys-e2e-explicit-4b0bf3-mjmdc\" (UID: \"f20d10be-ea2a-48ec-a877-64ffb303f56d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4b0bf3-mjmdc" Oct 30 20:23:34.076672 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:23:34.076638 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:23:34.108744 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:23:34.108658 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\"" Oct 30 20:23:34.147883 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:23:34.147669 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f20d10be-ea2a-48ec-a877-64ffb303f56d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-4b0bf3-mjmdc\" (UID: \"f20d10be-ea2a-48ec-a877-64ffb303f56d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4b0bf3-mjmdc" Oct 30 20:23:34.147883 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:23:34.147756 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f20d10be-ea2a-48ec-a877-64ffb303f56d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-4b0bf3-mjmdc\" (UID: \"f20d10be-ea2a-48ec-a877-64ffb303f56d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4b0bf3-mjmdc" Oct 30 20:23:34.147883 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:23:34.147805 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6lw7\" (UniqueName: \"kubernetes.io/projected/f20d10be-ea2a-48ec-a877-64ffb303f56d-kube-api-access-q6lw7\") pod \"maas-api-revoke-keys-e2e-explicit-4b0bf3-mjmdc\" (UID: \"f20d10be-ea2a-48ec-a877-64ffb303f56d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4b0bf3-mjmdc" Oct 30 20:23:34.150689 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:23:34.150656 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f20d10be-ea2a-48ec-a877-64ffb303f56d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-4b0bf3-mjmdc\" (UID: \"f20d10be-ea2a-48ec-a877-64ffb303f56d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4b0bf3-mjmdc" Oct 30 20:23:34.167218 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:23:34.167179 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f20d10be-ea2a-48ec-a877-64ffb303f56d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-4b0bf3-mjmdc\" (UID: \"f20d10be-ea2a-48ec-a877-64ffb303f56d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4b0bf3-mjmdc" Oct 30 20:23:34.212448 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:23:34.212419 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6lw7\" (UniqueName: \"kubernetes.io/projected/f20d10be-ea2a-48ec-a877-64ffb303f56d-kube-api-access-q6lw7\") pod \"maas-api-revoke-keys-e2e-explicit-4b0bf3-mjmdc\" (UID: \"f20d10be-ea2a-48ec-a877-64ffb303f56d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4b0bf3-mjmdc" Oct 30 20:23:34.285686 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:23:34.285655 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4b0bf3-mjmdc" Oct 30 20:23:34.667822 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:23:34.667796 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4b0bf3-mjmdc"] Oct 30 20:23:34.669665 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:23:34.669633 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf20d10be_ea2a_48ec_a877_64ffb303f56d.slice/crio-53dfe16a8b1df1fda1a0a6835a419f340825602519a7bfac3b82b7b4f54d0242 WatchSource:0}: Error finding container 53dfe16a8b1df1fda1a0a6835a419f340825602519a7bfac3b82b7b4f54d0242: Status 404 returned error can't find the container with id 53dfe16a8b1df1fda1a0a6835a419f340825602519a7bfac3b82b7b4f54d0242 Oct 30 20:23:35.155478 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:23:35.155448 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4b0bf3-mjmdc" event={"ID":"f20d10be-ea2a-48ec-a877-64ffb303f56d","Type":"ContainerStarted","Data":"a45731d1a42e3bc053db21d22a577915714f643f308769089648dd39c9e710cc"} Oct 30 20:23:35.155871 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:23:35.155497 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4b0bf3-mjmdc" event={"ID":"f20d10be-ea2a-48ec-a877-64ffb303f56d","Type":"ContainerStarted","Data":"53dfe16a8b1df1fda1a0a6835a419f340825602519a7bfac3b82b7b4f54d0242"} Oct 30 20:23:35.282488 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:23:35.282424 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4b0bf3-mjmdc" podStartSLOduration=2.282407464 podStartE2EDuration="2.282407464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:23:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:23:35.281963603 +0000 UTC m=+3518.800279060" watchObservedRunningTime="2026-10-30 20:23:35.282407464 +0000 UTC m=+3518.800722924" Oct 30 20:23:36.163153 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:23:36.163115 4292 generic.go:356] "Generic (PLEG): container finished" podID="f20d10be-ea2a-48ec-a877-64ffb303f56d" containerID="a45731d1a42e3bc053db21d22a577915714f643f308769089648dd39c9e710cc" exitCode=0 Oct 30 20:23:36.163612 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:23:36.163170 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4b0bf3-mjmdc" event={"ID":"f20d10be-ea2a-48ec-a877-64ffb303f56d","Type":"ContainerDied","Data":"a45731d1a42e3bc053db21d22a577915714f643f308769089648dd39c9e710cc"} Oct 30 20:23:37.325951 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:23:37.325925 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4b0bf3-mjmdc" Oct 30 20:23:37.390070 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:23:37.390042 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f20d10be-ea2a-48ec-a877-64ffb303f56d-kube-api-access-q6lw7\" (UniqueName: \"kubernetes.io/projected/f20d10be-ea2a-48ec-a877-64ffb303f56d-kube-api-access-q6lw7\") pod \"f20d10be-ea2a-48ec-a877-64ffb303f56d\" (UID: \"f20d10be-ea2a-48ec-a877-64ffb303f56d\") " Oct 30 20:23:37.390252 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:23:37.390114 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f20d10be-ea2a-48ec-a877-64ffb303f56d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f20d10be-ea2a-48ec-a877-64ffb303f56d-maas-api-openshift-ca\") pod \"f20d10be-ea2a-48ec-a877-64ffb303f56d\" (UID: \"f20d10be-ea2a-48ec-a877-64ffb303f56d\") " Oct 30 20:23:37.390252 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:23:37.390144 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f20d10be-ea2a-48ec-a877-64ffb303f56d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f20d10be-ea2a-48ec-a877-64ffb303f56d-maas-api-xks-ca\") pod \"f20d10be-ea2a-48ec-a877-64ffb303f56d\" (UID: \"f20d10be-ea2a-48ec-a877-64ffb303f56d\") " Oct 30 20:23:37.390536 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:23:37.390513 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f20d10be-ea2a-48ec-a877-64ffb303f56d-maas-api-openshift-ca" pod "f20d10be-ea2a-48ec-a877-64ffb303f56d" (UID: "f20d10be-ea2a-48ec-a877-64ffb303f56d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:23:37.392140 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:23:37.392118 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f20d10be-ea2a-48ec-a877-64ffb303f56d-maas-api-xks-ca" pod "f20d10be-ea2a-48ec-a877-64ffb303f56d" (UID: "f20d10be-ea2a-48ec-a877-64ffb303f56d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:23:37.392232 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:23:37.392214 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f20d10be-ea2a-48ec-a877-64ffb303f56d-kube-api-access-q6lw7" pod "f20d10be-ea2a-48ec-a877-64ffb303f56d" (UID: "f20d10be-ea2a-48ec-a877-64ffb303f56d"). InnerVolumeSpecName "kube-api-access-q6lw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:23:37.492255 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:23:37.492176 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6lw7\" (UniqueName: \"kubernetes.io/projected/f20d10be-ea2a-48ec-a877-64ffb303f56d-kube-api-access-q6lw7\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:23:37.492255 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:23:37.492204 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f20d10be-ea2a-48ec-a877-64ffb303f56d-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:23:37.492255 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:23:37.492215 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f20d10be-ea2a-48ec-a877-64ffb303f56d-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:23:38.179730 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:23:38.179696 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4b0bf3-mjmdc" Oct 30 20:23:38.179908 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:23:38.179701 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4b0bf3-mjmdc" event={"ID":"f20d10be-ea2a-48ec-a877-64ffb303f56d","Type":"ContainerDied","Data":"53dfe16a8b1df1fda1a0a6835a419f340825602519a7bfac3b82b7b4f54d0242"} Oct 30 20:23:38.179908 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:23:38.179821 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53dfe16a8b1df1fda1a0a6835a419f340825602519a7bfac3b82b7b4f54d0242" Oct 30 20:23:43.220041 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:23:43.220004 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:23:57.791821 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:23:57.791791 4292 scope.go:122] "RemoveContainer" containerID="16240c9b18721fcefcbd1bb57e9fa28872555bc5b6f241536d95b094dc941b9b" Oct 30 20:24:06.495466 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:06.495437 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k"] Oct 30 20:24:06.495967 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:06.495653 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k" podUID="4ea3800f-f814-4036-b3f4-b7e3573c011b" containerName="istio-proxy" containerID="cri-o://3242952e1de0e1fa818747fa2bf87de56695f5622a5361187ad7dade0afc32fa" gracePeriod=30 Oct 30 20:24:06.546774 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:06.546741 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht"] Oct 30 20:24:06.547006 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:06.546964 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht" podUID="8f9dd92b-3ba4-4d87-8162-df97a30a48d0" containerName="istio-proxy" containerID="cri-o://4f7a5376f0f046577e9750c1028030d6002a9b911bacc4cdc3a383fd027af037" gracePeriod=30 Oct 30 20:24:06.825984 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:06.825953 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kr46d"] Oct 30 20:24:06.826163 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:06.826136 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kr46d" podUID="de3476ef-83a0-4ca2-9b76-227ee4b35d7c" containerName="istio-proxy" containerID="cri-o://2fb856ce0abe7e7894144321855bd955ee318aee06031b6870882f8eab86b69c" gracePeriod=30 Oct 30 20:24:10.585758 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:10.585674 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:24:10.585758 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:10.585739 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht" podUID="8f9dd92b-3ba4-4d87-8162-df97a30a48d0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:24:11.498463 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:11.498418 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:24:11.498651 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:11.498483 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k" podUID="4ea3800f-f814-4036-b3f4-b7e3573c011b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:24:11.820971 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:11.820946 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k" Oct 30 20:24:11.864565 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:11.864538 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht" Oct 30 20:24:11.910072 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:11.910041 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ea3800f-f814-4036-b3f4-b7e3573c011b-kube-api-access-565ht\" (UniqueName: \"kubernetes.io/projected/4ea3800f-f814-4036-b3f4-b7e3573c011b-kube-api-access-565ht\") pod \"4ea3800f-f814-4036-b3f4-b7e3573c011b\" (UID: \"4ea3800f-f814-4036-b3f4-b7e3573c011b\") " Oct 30 20:24:11.910238 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:11.910104 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ea3800f-f814-4036-b3f4-b7e3573c011b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ea3800f-f814-4036-b3f4-b7e3573c011b-workload-certs\") pod \"4ea3800f-f814-4036-b3f4-b7e3573c011b\" (UID: \"4ea3800f-f814-4036-b3f4-b7e3573c011b\") " Oct 30 20:24:11.910238 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:11.910133 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ea3800f-f814-4036-b3f4-b7e3573c011b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ea3800f-f814-4036-b3f4-b7e3573c011b-istiod-ca-cert\") pod \"4ea3800f-f814-4036-b3f4-b7e3573c011b\" (UID: \"4ea3800f-f814-4036-b3f4-b7e3573c011b\") " Oct 30 20:24:11.910238 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:11.910204 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ea3800f-f814-4036-b3f4-b7e3573c011b-istio-token\" (UniqueName: \"kubernetes.io/projected/4ea3800f-f814-4036-b3f4-b7e3573c011b-istio-token\") pod \"4ea3800f-f814-4036-b3f4-b7e3573c011b\" (UID: \"4ea3800f-f814-4036-b3f4-b7e3573c011b\") " Oct 30 20:24:11.910864 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:11.910240 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ea3800f-f814-4036-b3f4-b7e3573c011b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ea3800f-f814-4036-b3f4-b7e3573c011b-istio-data\") pod \"4ea3800f-f814-4036-b3f4-b7e3573c011b\" (UID: \"4ea3800f-f814-4036-b3f4-b7e3573c011b\") " Oct 30 20:24:11.910864 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:11.910274 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ea3800f-f814-4036-b3f4-b7e3573c011b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ea3800f-f814-4036-b3f4-b7e3573c011b-workload-socket\") pod \"4ea3800f-f814-4036-b3f4-b7e3573c011b\" (UID: \"4ea3800f-f814-4036-b3f4-b7e3573c011b\") " Oct 30 20:24:11.910864 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:11.910321 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ea3800f-f814-4036-b3f4-b7e3573c011b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ea3800f-f814-4036-b3f4-b7e3573c011b-istio-envoy\") pod \"4ea3800f-f814-4036-b3f4-b7e3573c011b\" (UID: \"4ea3800f-f814-4036-b3f4-b7e3573c011b\") " Oct 30 20:24:11.910864 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:11.910389 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ea3800f-f814-4036-b3f4-b7e3573c011b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ea3800f-f814-4036-b3f4-b7e3573c011b-credential-socket\") pod \"4ea3800f-f814-4036-b3f4-b7e3573c011b\" (UID: \"4ea3800f-f814-4036-b3f4-b7e3573c011b\") " Oct 30 20:24:11.910864 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:11.910438 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ea3800f-f814-4036-b3f4-b7e3573c011b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ea3800f-f814-4036-b3f4-b7e3573c011b-istio-podinfo\") pod \"4ea3800f-f814-4036-b3f4-b7e3573c011b\" (UID: \"4ea3800f-f814-4036-b3f4-b7e3573c011b\") " Oct 30 20:24:11.910864 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:11.910435 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ea3800f-f814-4036-b3f4-b7e3573c011b-workload-certs" pod "4ea3800f-f814-4036-b3f4-b7e3573c011b" (UID: "4ea3800f-f814-4036-b3f4-b7e3573c011b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:24:11.910864 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:11.910478 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ea3800f-f814-4036-b3f4-b7e3573c011b-istio-data" pod "4ea3800f-f814-4036-b3f4-b7e3573c011b" (UID: "4ea3800f-f814-4036-b3f4-b7e3573c011b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:24:11.910864 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:11.910577 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ea3800f-f814-4036-b3f4-b7e3573c011b-workload-socket" pod "4ea3800f-f814-4036-b3f4-b7e3573c011b" (UID: "4ea3800f-f814-4036-b3f4-b7e3573c011b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:24:11.910864 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:11.910605 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ea3800f-f814-4036-b3f4-b7e3573c011b-istiod-ca-cert" pod "4ea3800f-f814-4036-b3f4-b7e3573c011b" (UID: "4ea3800f-f814-4036-b3f4-b7e3573c011b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:24:11.910864 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:11.910730 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ea3800f-f814-4036-b3f4-b7e3573c011b-credential-socket" pod "4ea3800f-f814-4036-b3f4-b7e3573c011b" (UID: "4ea3800f-f814-4036-b3f4-b7e3573c011b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:24:11.911394 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:11.910927 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ea3800f-f814-4036-b3f4-b7e3573c011b-credential-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:24:11.911394 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:11.910954 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ea3800f-f814-4036-b3f4-b7e3573c011b-workload-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:24:11.911394 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:11.910969 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ea3800f-f814-4036-b3f4-b7e3573c011b-istiod-ca-cert\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:24:11.911394 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:11.910982 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ea3800f-f814-4036-b3f4-b7e3573c011b-istio-data\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:24:11.911394 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:11.910995 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ea3800f-f814-4036-b3f4-b7e3573c011b-workload-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:24:11.912762 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:11.912741 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ea3800f-f814-4036-b3f4-b7e3573c011b-kube-api-access-565ht" pod "4ea3800f-f814-4036-b3f4-b7e3573c011b" (UID: "4ea3800f-f814-4036-b3f4-b7e3573c011b"). InnerVolumeSpecName "kube-api-access-565ht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:24:11.912967 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:11.912952 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ea3800f-f814-4036-b3f4-b7e3573c011b-istio-token" pod "4ea3800f-f814-4036-b3f4-b7e3573c011b" (UID: "4ea3800f-f814-4036-b3f4-b7e3573c011b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:24:11.913018 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:11.913002 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ea3800f-f814-4036-b3f4-b7e3573c011b-istio-envoy" pod "4ea3800f-f814-4036-b3f4-b7e3573c011b" (UID: "4ea3800f-f814-4036-b3f4-b7e3573c011b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:24:11.913069 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:11.913055 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ea3800f-f814-4036-b3f4-b7e3573c011b-istio-podinfo" pod "4ea3800f-f814-4036-b3f4-b7e3573c011b" (UID: "4ea3800f-f814-4036-b3f4-b7e3573c011b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:24:12.013247 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.012573 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f9dd92b-3ba4-4d87-8162-df97a30a48d0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f9dd92b-3ba4-4d87-8162-df97a30a48d0-workload-socket\") pod \"8f9dd92b-3ba4-4d87-8162-df97a30a48d0\" (UID: \"8f9dd92b-3ba4-4d87-8162-df97a30a48d0\") " Oct 30 20:24:12.013247 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.012635 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f9dd92b-3ba4-4d87-8162-df97a30a48d0-istio-token\" (UniqueName: \"kubernetes.io/projected/8f9dd92b-3ba4-4d87-8162-df97a30a48d0-istio-token\") pod \"8f9dd92b-3ba4-4d87-8162-df97a30a48d0\" (UID: \"8f9dd92b-3ba4-4d87-8162-df97a30a48d0\") " Oct 30 20:24:12.013247 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.012665 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f9dd92b-3ba4-4d87-8162-df97a30a48d0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f9dd92b-3ba4-4d87-8162-df97a30a48d0-istiod-ca-cert\") pod \"8f9dd92b-3ba4-4d87-8162-df97a30a48d0\" (UID: \"8f9dd92b-3ba4-4d87-8162-df97a30a48d0\") " Oct 30 20:24:12.013247 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.012750 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f9dd92b-3ba4-4d87-8162-df97a30a48d0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f9dd92b-3ba4-4d87-8162-df97a30a48d0-istio-envoy\") pod \"8f9dd92b-3ba4-4d87-8162-df97a30a48d0\" (UID: \"8f9dd92b-3ba4-4d87-8162-df97a30a48d0\") " Oct 30 20:24:12.013247 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.012796 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8f9dd92b-3ba4-4d87-8162-df97a30a48d0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f9dd92b-3ba4-4d87-8162-df97a30a48d0-istio-podinfo\") pod \"8f9dd92b-3ba4-4d87-8162-df97a30a48d0\" (UID: \"8f9dd92b-3ba4-4d87-8162-df97a30a48d0\") " Oct 30 20:24:12.013247 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.012873 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f9dd92b-3ba4-4d87-8162-df97a30a48d0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f9dd92b-3ba4-4d87-8162-df97a30a48d0-credential-socket\") pod \"8f9dd92b-3ba4-4d87-8162-df97a30a48d0\" (UID: \"8f9dd92b-3ba4-4d87-8162-df97a30a48d0\") " Oct 30 20:24:12.013247 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.012901 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f9dd92b-3ba4-4d87-8162-df97a30a48d0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f9dd92b-3ba4-4d87-8162-df97a30a48d0-workload-certs\") pod \"8f9dd92b-3ba4-4d87-8162-df97a30a48d0\" (UID: \"8f9dd92b-3ba4-4d87-8162-df97a30a48d0\") " Oct 30 20:24:12.013247 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.012954 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f9dd92b-3ba4-4d87-8162-df97a30a48d0-kube-api-access-q6ftt\" (UniqueName: \"kubernetes.io/projected/8f9dd92b-3ba4-4d87-8162-df97a30a48d0-kube-api-access-q6ftt\") pod \"8f9dd92b-3ba4-4d87-8162-df97a30a48d0\" (UID: \"8f9dd92b-3ba4-4d87-8162-df97a30a48d0\") " Oct 30 20:24:12.013247 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.013065 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f9dd92b-3ba4-4d87-8162-df97a30a48d0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f9dd92b-3ba4-4d87-8162-df97a30a48d0-istio-data\") pod \"8f9dd92b-3ba4-4d87-8162-df97a30a48d0\" (UID: \"8f9dd92b-3ba4-4d87-8162-df97a30a48d0\") " Oct 30 20:24:12.016149 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.015133 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ea3800f-f814-4036-b3f4-b7e3573c011b-istio-token\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:24:12.016149 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.015168 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ea3800f-f814-4036-b3f4-b7e3573c011b-istio-envoy\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:24:12.016149 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.015185 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ea3800f-f814-4036-b3f4-b7e3573c011b-istio-podinfo\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:24:12.016149 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.015201 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-565ht\" (UniqueName: \"kubernetes.io/projected/4ea3800f-f814-4036-b3f4-b7e3573c011b-kube-api-access-565ht\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:24:12.017733 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.016590 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f9dd92b-3ba4-4d87-8162-df97a30a48d0-istio-data" pod "8f9dd92b-3ba4-4d87-8162-df97a30a48d0" (UID: "8f9dd92b-3ba4-4d87-8162-df97a30a48d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:24:12.019468 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.019369 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f9dd92b-3ba4-4d87-8162-df97a30a48d0-credential-socket" pod "8f9dd92b-3ba4-4d87-8162-df97a30a48d0" (UID: "8f9dd92b-3ba4-4d87-8162-df97a30a48d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:24:12.019663 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.019641 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f9dd92b-3ba4-4d87-8162-df97a30a48d0-workload-socket" pod "8f9dd92b-3ba4-4d87-8162-df97a30a48d0" (UID: "8f9dd92b-3ba4-4d87-8162-df97a30a48d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:24:12.019961 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.019936 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f9dd92b-3ba4-4d87-8162-df97a30a48d0-istiod-ca-cert" pod "8f9dd92b-3ba4-4d87-8162-df97a30a48d0" (UID: "8f9dd92b-3ba4-4d87-8162-df97a30a48d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:24:12.020446 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.020292 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f9dd92b-3ba4-4d87-8162-df97a30a48d0-workload-certs" pod "8f9dd92b-3ba4-4d87-8162-df97a30a48d0" (UID: "8f9dd92b-3ba4-4d87-8162-df97a30a48d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:24:12.022841 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.022819 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f9dd92b-3ba4-4d87-8162-df97a30a48d0-istio-envoy" pod "8f9dd92b-3ba4-4d87-8162-df97a30a48d0" (UID: "8f9dd92b-3ba4-4d87-8162-df97a30a48d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:24:12.022948 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.022863 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f9dd92b-3ba4-4d87-8162-df97a30a48d0-kube-api-access-q6ftt" pod "8f9dd92b-3ba4-4d87-8162-df97a30a48d0" (UID: "8f9dd92b-3ba4-4d87-8162-df97a30a48d0"). InnerVolumeSpecName "kube-api-access-q6ftt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:24:12.023340 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.023315 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f9dd92b-3ba4-4d87-8162-df97a30a48d0-istio-podinfo" pod "8f9dd92b-3ba4-4d87-8162-df97a30a48d0" (UID: "8f9dd92b-3ba4-4d87-8162-df97a30a48d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:24:12.023491 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.023423 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f9dd92b-3ba4-4d87-8162-df97a30a48d0-istio-token" pod "8f9dd92b-3ba4-4d87-8162-df97a30a48d0" (UID: "8f9dd92b-3ba4-4d87-8162-df97a30a48d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:24:12.080255 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.080230 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kr46d" Oct 30 20:24:12.116599 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.116568 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f9dd92b-3ba4-4d87-8162-df97a30a48d0-istio-data\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:24:12.116599 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.116602 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f9dd92b-3ba4-4d87-8162-df97a30a48d0-workload-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:24:12.116778 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.116619 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f9dd92b-3ba4-4d87-8162-df97a30a48d0-istio-token\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:24:12.116778 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.116633 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f9dd92b-3ba4-4d87-8162-df97a30a48d0-istiod-ca-cert\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:24:12.116778 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.116647 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f9dd92b-3ba4-4d87-8162-df97a30a48d0-istio-envoy\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:24:12.116778 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.116661 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f9dd92b-3ba4-4d87-8162-df97a30a48d0-istio-podinfo\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:24:12.116778 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.116675 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f9dd92b-3ba4-4d87-8162-df97a30a48d0-credential-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:24:12.116778 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.116690 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f9dd92b-3ba4-4d87-8162-df97a30a48d0-workload-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:24:12.116778 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.116703 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6ftt\" (UniqueName: \"kubernetes.io/projected/8f9dd92b-3ba4-4d87-8162-df97a30a48d0-kube-api-access-q6ftt\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:24:12.217921 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.217886 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de3476ef-83a0-4ca2-9b76-227ee4b35d7c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de3476ef-83a0-4ca2-9b76-227ee4b35d7c-workload-certs\") pod \"de3476ef-83a0-4ca2-9b76-227ee4b35d7c\" (UID: \"de3476ef-83a0-4ca2-9b76-227ee4b35d7c\") " Oct 30 20:24:12.218099 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.217933 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de3476ef-83a0-4ca2-9b76-227ee4b35d7c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de3476ef-83a0-4ca2-9b76-227ee4b35d7c-istiod-ca-cert\") pod \"de3476ef-83a0-4ca2-9b76-227ee4b35d7c\" (UID: \"de3476ef-83a0-4ca2-9b76-227ee4b35d7c\") " Oct 30 20:24:12.218099 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.217963 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de3476ef-83a0-4ca2-9b76-227ee4b35d7c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de3476ef-83a0-4ca2-9b76-227ee4b35d7c-istio-envoy\") pod \"de3476ef-83a0-4ca2-9b76-227ee4b35d7c\" (UID: \"de3476ef-83a0-4ca2-9b76-227ee4b35d7c\") " Oct 30 20:24:12.218099 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.218061 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de3476ef-83a0-4ca2-9b76-227ee4b35d7c-istio-token\" (UniqueName: \"kubernetes.io/projected/de3476ef-83a0-4ca2-9b76-227ee4b35d7c-istio-token\") pod \"de3476ef-83a0-4ca2-9b76-227ee4b35d7c\" (UID: \"de3476ef-83a0-4ca2-9b76-227ee4b35d7c\") " Oct 30 20:24:12.218308 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.218132 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de3476ef-83a0-4ca2-9b76-227ee4b35d7c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de3476ef-83a0-4ca2-9b76-227ee4b35d7c-istio-data\") pod \"de3476ef-83a0-4ca2-9b76-227ee4b35d7c\" (UID: \"de3476ef-83a0-4ca2-9b76-227ee4b35d7c\") " Oct 30 20:24:12.218308 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.218188 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de3476ef-83a0-4ca2-9b76-227ee4b35d7c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de3476ef-83a0-4ca2-9b76-227ee4b35d7c-workload-socket\") pod \"de3476ef-83a0-4ca2-9b76-227ee4b35d7c\" (UID: \"de3476ef-83a0-4ca2-9b76-227ee4b35d7c\") " Oct 30 20:24:12.218308 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.218208 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de3476ef-83a0-4ca2-9b76-227ee4b35d7c-workload-certs" pod "de3476ef-83a0-4ca2-9b76-227ee4b35d7c" (UID: "de3476ef-83a0-4ca2-9b76-227ee4b35d7c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:24:12.218308 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.218261 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de3476ef-83a0-4ca2-9b76-227ee4b35d7c-kube-api-access-79l4d\" (UniqueName: \"kubernetes.io/projected/de3476ef-83a0-4ca2-9b76-227ee4b35d7c-kube-api-access-79l4d\") pod \"de3476ef-83a0-4ca2-9b76-227ee4b35d7c\" (UID: \"de3476ef-83a0-4ca2-9b76-227ee4b35d7c\") " Oct 30 20:24:12.218308 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.218295 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de3476ef-83a0-4ca2-9b76-227ee4b35d7c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de3476ef-83a0-4ca2-9b76-227ee4b35d7c-istio-podinfo\") pod \"de3476ef-83a0-4ca2-9b76-227ee4b35d7c\" (UID: \"de3476ef-83a0-4ca2-9b76-227ee4b35d7c\") " Oct 30 20:24:12.218627 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.218373 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de3476ef-83a0-4ca2-9b76-227ee4b35d7c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de3476ef-83a0-4ca2-9b76-227ee4b35d7c-credential-socket\") pod \"de3476ef-83a0-4ca2-9b76-227ee4b35d7c\" (UID: \"de3476ef-83a0-4ca2-9b76-227ee4b35d7c\") " Oct 30 20:24:12.218627 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.218434 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de3476ef-83a0-4ca2-9b76-227ee4b35d7c-istiod-ca-cert" pod "de3476ef-83a0-4ca2-9b76-227ee4b35d7c" (UID: "de3476ef-83a0-4ca2-9b76-227ee4b35d7c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:24:12.218627 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.218462 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de3476ef-83a0-4ca2-9b76-227ee4b35d7c-istio-data" pod "de3476ef-83a0-4ca2-9b76-227ee4b35d7c" (UID: "de3476ef-83a0-4ca2-9b76-227ee4b35d7c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:24:12.218780 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.218699 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de3476ef-83a0-4ca2-9b76-227ee4b35d7c-workload-socket" pod "de3476ef-83a0-4ca2-9b76-227ee4b35d7c" (UID: "de3476ef-83a0-4ca2-9b76-227ee4b35d7c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:24:12.218878 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.218860 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de3476ef-83a0-4ca2-9b76-227ee4b35d7c-credential-socket" pod "de3476ef-83a0-4ca2-9b76-227ee4b35d7c" (UID: "de3476ef-83a0-4ca2-9b76-227ee4b35d7c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:24:12.218959 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.218928 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de3476ef-83a0-4ca2-9b76-227ee4b35d7c-workload-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:24:12.218959 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.218951 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de3476ef-83a0-4ca2-9b76-227ee4b35d7c-workload-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:24:12.219134 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.218964 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de3476ef-83a0-4ca2-9b76-227ee4b35d7c-istiod-ca-cert\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:24:12.219134 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.218988 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de3476ef-83a0-4ca2-9b76-227ee4b35d7c-istio-data\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:24:12.220535 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.220511 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de3476ef-83a0-4ca2-9b76-227ee4b35d7c-istio-envoy" pod "de3476ef-83a0-4ca2-9b76-227ee4b35d7c" (UID: "de3476ef-83a0-4ca2-9b76-227ee4b35d7c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:24:12.220748 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.220724 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de3476ef-83a0-4ca2-9b76-227ee4b35d7c-istio-podinfo" pod "de3476ef-83a0-4ca2-9b76-227ee4b35d7c" (UID: "de3476ef-83a0-4ca2-9b76-227ee4b35d7c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:24:12.220824 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.220758 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de3476ef-83a0-4ca2-9b76-227ee4b35d7c-istio-token" pod "de3476ef-83a0-4ca2-9b76-227ee4b35d7c" (UID: "de3476ef-83a0-4ca2-9b76-227ee4b35d7c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:24:12.220886 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.220874 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de3476ef-83a0-4ca2-9b76-227ee4b35d7c-kube-api-access-79l4d" pod "de3476ef-83a0-4ca2-9b76-227ee4b35d7c" (UID: "de3476ef-83a0-4ca2-9b76-227ee4b35d7c"). InnerVolumeSpecName "kube-api-access-79l4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:24:12.319741 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.319705 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79l4d\" (UniqueName: \"kubernetes.io/projected/de3476ef-83a0-4ca2-9b76-227ee4b35d7c-kube-api-access-79l4d\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:24:12.319741 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.319735 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de3476ef-83a0-4ca2-9b76-227ee4b35d7c-istio-podinfo\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:24:12.319741 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.319745 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de3476ef-83a0-4ca2-9b76-227ee4b35d7c-credential-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:24:12.319975 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.319754 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de3476ef-83a0-4ca2-9b76-227ee4b35d7c-istio-envoy\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:24:12.319975 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.319763 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de3476ef-83a0-4ca2-9b76-227ee4b35d7c-istio-token\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:24:12.375614 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.375580 4292 generic.go:356] "Generic (PLEG): container finished" podID="8f9dd92b-3ba4-4d87-8162-df97a30a48d0" containerID="4f7a5376f0f046577e9750c1028030d6002a9b911bacc4cdc3a383fd027af037" exitCode=0 Oct 30 20:24:12.375799 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.375653 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht" Oct 30 20:24:12.375799 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.375660 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht" event={"ID":"8f9dd92b-3ba4-4d87-8162-df97a30a48d0","Type":"ContainerDied","Data":"4f7a5376f0f046577e9750c1028030d6002a9b911bacc4cdc3a383fd027af037"} Oct 30 20:24:12.375799 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.375743 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht" event={"ID":"8f9dd92b-3ba4-4d87-8162-df97a30a48d0","Type":"ContainerDied","Data":"513e7ad811b4c198baa46c0390dd393155127179d2707603b70e1ed15f4fcdef"} Oct 30 20:24:12.375799 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.375772 4292 scope.go:122] "RemoveContainer" containerID="4f7a5376f0f046577e9750c1028030d6002a9b911bacc4cdc3a383fd027af037" Oct 30 20:24:12.377229 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.377208 4292 generic.go:356] "Generic (PLEG): container finished" podID="de3476ef-83a0-4ca2-9b76-227ee4b35d7c" containerID="2fb856ce0abe7e7894144321855bd955ee318aee06031b6870882f8eab86b69c" exitCode=0 Oct 30 20:24:12.377396 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.377250 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kr46d" Oct 30 20:24:12.377396 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.377294 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kr46d" event={"ID":"de3476ef-83a0-4ca2-9b76-227ee4b35d7c","Type":"ContainerDied","Data":"2fb856ce0abe7e7894144321855bd955ee318aee06031b6870882f8eab86b69c"} Oct 30 20:24:12.377533 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.377399 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kr46d" event={"ID":"de3476ef-83a0-4ca2-9b76-227ee4b35d7c","Type":"ContainerDied","Data":"f34b6db23573cfeec4dbf1665c4c30e3f6d7745525c38e35c5ef8b1290683066"} Oct 30 20:24:12.396043 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.392529 4292 generic.go:356] "Generic (PLEG): container finished" podID="4ea3800f-f814-4036-b3f4-b7e3573c011b" containerID="3242952e1de0e1fa818747fa2bf87de56695f5622a5361187ad7dade0afc32fa" exitCode=0 Oct 30 20:24:12.396043 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.392628 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k" event={"ID":"4ea3800f-f814-4036-b3f4-b7e3573c011b","Type":"ContainerDied","Data":"3242952e1de0e1fa818747fa2bf87de56695f5622a5361187ad7dade0afc32fa"} Oct 30 20:24:12.396043 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.392678 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k" event={"ID":"4ea3800f-f814-4036-b3f4-b7e3573c011b","Type":"ContainerDied","Data":"5584dd58fea5014e2d83638d8390f90f96977e165471ea54a68deff5538c0f9d"} Oct 30 20:24:12.396043 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.392679 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k" Oct 30 20:24:12.438405 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.438385 4292 scope.go:122] "RemoveContainer" containerID="4f7a5376f0f046577e9750c1028030d6002a9b911bacc4cdc3a383fd027af037" Oct 30 20:24:12.438791 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:24:12.438769 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f7a5376f0f046577e9750c1028030d6002a9b911bacc4cdc3a383fd027af037\": container with ID starting with 4f7a5376f0f046577e9750c1028030d6002a9b911bacc4cdc3a383fd027af037 not found: ID does not exist" containerID="4f7a5376f0f046577e9750c1028030d6002a9b911bacc4cdc3a383fd027af037" Oct 30 20:24:12.438868 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.438805 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f7a5376f0f046577e9750c1028030d6002a9b911bacc4cdc3a383fd027af037"} err="failed to get container status \"4f7a5376f0f046577e9750c1028030d6002a9b911bacc4cdc3a383fd027af037\": rpc error: code = NotFound desc = could not find container \"4f7a5376f0f046577e9750c1028030d6002a9b911bacc4cdc3a383fd027af037\": container with ID starting with 4f7a5376f0f046577e9750c1028030d6002a9b911bacc4cdc3a383fd027af037 not found: ID does not exist" Oct 30 20:24:12.438868 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.438829 4292 scope.go:122] "RemoveContainer" containerID="2fb856ce0abe7e7894144321855bd955ee318aee06031b6870882f8eab86b69c" Oct 30 20:24:12.447805 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.447786 4292 scope.go:122] "RemoveContainer" containerID="2fb856ce0abe7e7894144321855bd955ee318aee06031b6870882f8eab86b69c" Oct 30 20:24:12.448102 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:24:12.448075 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fb856ce0abe7e7894144321855bd955ee318aee06031b6870882f8eab86b69c\": container with ID starting with 2fb856ce0abe7e7894144321855bd955ee318aee06031b6870882f8eab86b69c not found: ID does not exist" containerID="2fb856ce0abe7e7894144321855bd955ee318aee06031b6870882f8eab86b69c" Oct 30 20:24:12.448180 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.448117 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fb856ce0abe7e7894144321855bd955ee318aee06031b6870882f8eab86b69c"} err="failed to get container status \"2fb856ce0abe7e7894144321855bd955ee318aee06031b6870882f8eab86b69c\": rpc error: code = NotFound desc = could not find container \"2fb856ce0abe7e7894144321855bd955ee318aee06031b6870882f8eab86b69c\": container with ID starting with 2fb856ce0abe7e7894144321855bd955ee318aee06031b6870882f8eab86b69c not found: ID does not exist" Oct 30 20:24:12.448180 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.448142 4292 scope.go:122] "RemoveContainer" containerID="3242952e1de0e1fa818747fa2bf87de56695f5622a5361187ad7dade0afc32fa" Oct 30 20:24:12.456740 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.456700 4292 scope.go:122] "RemoveContainer" containerID="3242952e1de0e1fa818747fa2bf87de56695f5622a5361187ad7dade0afc32fa" Oct 30 20:24:12.456980 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:24:12.456952 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3242952e1de0e1fa818747fa2bf87de56695f5622a5361187ad7dade0afc32fa\": container with ID starting with 3242952e1de0e1fa818747fa2bf87de56695f5622a5361187ad7dade0afc32fa not found: ID does not exist" containerID="3242952e1de0e1fa818747fa2bf87de56695f5622a5361187ad7dade0afc32fa" Oct 30 20:24:12.457026 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.456989 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3242952e1de0e1fa818747fa2bf87de56695f5622a5361187ad7dade0afc32fa"} err="failed to get container status \"3242952e1de0e1fa818747fa2bf87de56695f5622a5361187ad7dade0afc32fa\": rpc error: code = NotFound desc = could not find container \"3242952e1de0e1fa818747fa2bf87de56695f5622a5361187ad7dade0afc32fa\": container with ID starting with 3242952e1de0e1fa818747fa2bf87de56695f5622a5361187ad7dade0afc32fa not found: ID does not exist" Oct 30 20:24:12.545982 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.545950 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht"] Oct 30 20:24:12.551466 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.551442 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-n4xht"] Oct 30 20:24:12.582232 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.582171 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f9dd92b-3ba4-4d87-8162-df97a30a48d0" path="/var/lib/kubelet/pods/8f9dd92b-3ba4-4d87-8162-df97a30a48d0/volumes" Oct 30 20:24:12.663572 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.663507 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k"] Oct 30 20:24:12.666212 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.666189 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-knf7k"] Oct 30 20:24:12.779481 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.779452 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kr46d"] Oct 30 20:24:12.783823 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:12.783799 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-kr46d"] Oct 30 20:24:13.219933 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:24:13.219902 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:24:13.245029 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:13.244998 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-64120b/auth-test-model-64120b-kserve-8968677fc-q4rsd"] Oct 30 20:24:13.245503 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:13.245489 4292 cpu_manager.go:414] "containerMap: removing container" podUID="f20d10be-ea2a-48ec-a877-64ffb303f56d" containerName="revoke-keys" Oct 30 20:24:13.245555 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:13.245504 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f20d10be-ea2a-48ec-a877-64ffb303f56d" containerName="revoke-keys" Oct 30 20:24:13.245555 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:13.245534 4292 cpu_manager.go:414] "containerMap: removing container" podUID="4ea3800f-f814-4036-b3f4-b7e3573c011b" containerName="istio-proxy" Oct 30 20:24:13.245555 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:13.245540 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ea3800f-f814-4036-b3f4-b7e3573c011b" containerName="istio-proxy" Oct 30 20:24:13.245555 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:13.245550 4292 cpu_manager.go:414] "containerMap: removing container" podUID="de3476ef-83a0-4ca2-9b76-227ee4b35d7c" containerName="istio-proxy" Oct 30 20:24:13.245555 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:13.245555 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de3476ef-83a0-4ca2-9b76-227ee4b35d7c" containerName="istio-proxy" Oct 30 20:24:13.245716 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:13.245568 4292 cpu_manager.go:414] "containerMap: removing container" podUID="8f9dd92b-3ba4-4d87-8162-df97a30a48d0" containerName="istio-proxy" Oct 30 20:24:13.245716 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:13.245572 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f9dd92b-3ba4-4d87-8162-df97a30a48d0" containerName="istio-proxy" Oct 30 20:24:13.245716 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:13.245642 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f9dd92b-3ba4-4d87-8162-df97a30a48d0" containerName="istio-proxy" Oct 30 20:24:13.245716 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:13.245651 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="f20d10be-ea2a-48ec-a877-64ffb303f56d" containerName="revoke-keys" Oct 30 20:24:13.245716 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:13.245660 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ea3800f-f814-4036-b3f4-b7e3573c011b" containerName="istio-proxy" Oct 30 20:24:13.245716 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:13.245668 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="de3476ef-83a0-4ca2-9b76-227ee4b35d7c" containerName="istio-proxy" Oct 30 20:24:13.250494 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:13.250477 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-64120b/auth-test-model-64120b-kserve-8968677fc-q4rsd" Oct 30 20:24:13.304362 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:13.304322 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-64120b\"/\"kube-root-ca.crt\"" Oct 30 20:24:13.305658 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:13.305558 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-64120b/auth-test-model-64120b-kserve-8968677fc-q4rsd"] Oct 30 20:24:13.330660 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:13.330634 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f9c4f3a-c93c-4315-b0c0-95131a3413b6-home\") pod \"auth-test-model-64120b-kserve-8968677fc-q4rsd\" (UID: \"8f9c4f3a-c93c-4315-b0c0-95131a3413b6\") " pod="ai-tenant-e2e-shared-b-w2-64120b/auth-test-model-64120b-kserve-8968677fc-q4rsd" Oct 30 20:24:13.330824 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:13.330680 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f9c4f3a-c93c-4315-b0c0-95131a3413b6-model-cache\") pod \"auth-test-model-64120b-kserve-8968677fc-q4rsd\" (UID: \"8f9c4f3a-c93c-4315-b0c0-95131a3413b6\") " pod="ai-tenant-e2e-shared-b-w2-64120b/auth-test-model-64120b-kserve-8968677fc-q4rsd" Oct 30 20:24:13.330900 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:13.330817 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f9c4f3a-c93c-4315-b0c0-95131a3413b6-tls-certs\") pod \"auth-test-model-64120b-kserve-8968677fc-q4rsd\" (UID: \"8f9c4f3a-c93c-4315-b0c0-95131a3413b6\") " pod="ai-tenant-e2e-shared-b-w2-64120b/auth-test-model-64120b-kserve-8968677fc-q4rsd" Oct 30 20:24:13.330961 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:13.330942 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ltwn\" (UniqueName: \"kubernetes.io/projected/8f9c4f3a-c93c-4315-b0c0-95131a3413b6-kube-api-access-4ltwn\") pod \"auth-test-model-64120b-kserve-8968677fc-q4rsd\" (UID: \"8f9c4f3a-c93c-4315-b0c0-95131a3413b6\") " pod="ai-tenant-e2e-shared-b-w2-64120b/auth-test-model-64120b-kserve-8968677fc-q4rsd" Oct 30 20:24:13.331015 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:13.330996 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f9c4f3a-c93c-4315-b0c0-95131a3413b6-tmp-dir\") pod \"auth-test-model-64120b-kserve-8968677fc-q4rsd\" (UID: \"8f9c4f3a-c93c-4315-b0c0-95131a3413b6\") " pod="ai-tenant-e2e-shared-b-w2-64120b/auth-test-model-64120b-kserve-8968677fc-q4rsd" Oct 30 20:24:13.331071 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:13.331059 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f9c4f3a-c93c-4315-b0c0-95131a3413b6-dshm\") pod \"auth-test-model-64120b-kserve-8968677fc-q4rsd\" (UID: \"8f9c4f3a-c93c-4315-b0c0-95131a3413b6\") " pod="ai-tenant-e2e-shared-b-w2-64120b/auth-test-model-64120b-kserve-8968677fc-q4rsd" Oct 30 20:24:13.352363 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:13.352337 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-64120b\"/\"default-dockercfg-qnqfn\"" Oct 30 20:24:13.352479 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:13.352341 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-64120b\"/\"openshift-service-ca.crt\"" Oct 30 20:24:13.352479 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:13.352385 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-64120b\"/\"auth-test-model-64120b-kserve-self-signed-certs\"" Oct 30 20:24:13.432416 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:13.432387 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f9c4f3a-c93c-4315-b0c0-95131a3413b6-home\") pod \"auth-test-model-64120b-kserve-8968677fc-q4rsd\" (UID: \"8f9c4f3a-c93c-4315-b0c0-95131a3413b6\") " pod="ai-tenant-e2e-shared-b-w2-64120b/auth-test-model-64120b-kserve-8968677fc-q4rsd" Oct 30 20:24:13.432617 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:13.432508 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f9c4f3a-c93c-4315-b0c0-95131a3413b6-model-cache\") pod \"auth-test-model-64120b-kserve-8968677fc-q4rsd\" (UID: \"8f9c4f3a-c93c-4315-b0c0-95131a3413b6\") " pod="ai-tenant-e2e-shared-b-w2-64120b/auth-test-model-64120b-kserve-8968677fc-q4rsd" Oct 30 20:24:13.432617 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:13.432558 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f9c4f3a-c93c-4315-b0c0-95131a3413b6-tls-certs\") pod \"auth-test-model-64120b-kserve-8968677fc-q4rsd\" (UID: \"8f9c4f3a-c93c-4315-b0c0-95131a3413b6\") " pod="ai-tenant-e2e-shared-b-w2-64120b/auth-test-model-64120b-kserve-8968677fc-q4rsd" Oct 30 20:24:13.432617 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:13.432616 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ltwn\" (UniqueName: \"kubernetes.io/projected/8f9c4f3a-c93c-4315-b0c0-95131a3413b6-kube-api-access-4ltwn\") pod \"auth-test-model-64120b-kserve-8968677fc-q4rsd\" (UID: \"8f9c4f3a-c93c-4315-b0c0-95131a3413b6\") " pod="ai-tenant-e2e-shared-b-w2-64120b/auth-test-model-64120b-kserve-8968677fc-q4rsd" Oct 30 20:24:13.432832 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:13.432637 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f9c4f3a-c93c-4315-b0c0-95131a3413b6-tmp-dir\") pod \"auth-test-model-64120b-kserve-8968677fc-q4rsd\" (UID: \"8f9c4f3a-c93c-4315-b0c0-95131a3413b6\") " pod="ai-tenant-e2e-shared-b-w2-64120b/auth-test-model-64120b-kserve-8968677fc-q4rsd" Oct 30 20:24:13.432832 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:13.432675 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f9c4f3a-c93c-4315-b0c0-95131a3413b6-dshm\") pod \"auth-test-model-64120b-kserve-8968677fc-q4rsd\" (UID: \"8f9c4f3a-c93c-4315-b0c0-95131a3413b6\") " pod="ai-tenant-e2e-shared-b-w2-64120b/auth-test-model-64120b-kserve-8968677fc-q4rsd" Oct 30 20:24:13.432935 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:13.432828 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f9c4f3a-c93c-4315-b0c0-95131a3413b6-home\") pod \"auth-test-model-64120b-kserve-8968677fc-q4rsd\" (UID: \"8f9c4f3a-c93c-4315-b0c0-95131a3413b6\") " pod="ai-tenant-e2e-shared-b-w2-64120b/auth-test-model-64120b-kserve-8968677fc-q4rsd" Oct 30 20:24:13.432935 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:13.432889 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f9c4f3a-c93c-4315-b0c0-95131a3413b6-model-cache\") pod \"auth-test-model-64120b-kserve-8968677fc-q4rsd\" (UID: \"8f9c4f3a-c93c-4315-b0c0-95131a3413b6\") " pod="ai-tenant-e2e-shared-b-w2-64120b/auth-test-model-64120b-kserve-8968677fc-q4rsd" Oct 30 20:24:13.433001 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:13.432954 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f9c4f3a-c93c-4315-b0c0-95131a3413b6-tmp-dir\") pod \"auth-test-model-64120b-kserve-8968677fc-q4rsd\" (UID: \"8f9c4f3a-c93c-4315-b0c0-95131a3413b6\") " pod="ai-tenant-e2e-shared-b-w2-64120b/auth-test-model-64120b-kserve-8968677fc-q4rsd" Oct 30 20:24:13.434882 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:13.434859 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f9c4f3a-c93c-4315-b0c0-95131a3413b6-dshm\") pod \"auth-test-model-64120b-kserve-8968677fc-q4rsd\" (UID: \"8f9c4f3a-c93c-4315-b0c0-95131a3413b6\") " pod="ai-tenant-e2e-shared-b-w2-64120b/auth-test-model-64120b-kserve-8968677fc-q4rsd" Oct 30 20:24:13.435196 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:13.435180 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f9c4f3a-c93c-4315-b0c0-95131a3413b6-tls-certs\") pod \"auth-test-model-64120b-kserve-8968677fc-q4rsd\" (UID: \"8f9c4f3a-c93c-4315-b0c0-95131a3413b6\") " pod="ai-tenant-e2e-shared-b-w2-64120b/auth-test-model-64120b-kserve-8968677fc-q4rsd" Oct 30 20:24:13.495585 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:13.495519 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ltwn\" (UniqueName: \"kubernetes.io/projected/8f9c4f3a-c93c-4315-b0c0-95131a3413b6-kube-api-access-4ltwn\") pod \"auth-test-model-64120b-kserve-8968677fc-q4rsd\" (UID: \"8f9c4f3a-c93c-4315-b0c0-95131a3413b6\") " pod="ai-tenant-e2e-shared-b-w2-64120b/auth-test-model-64120b-kserve-8968677fc-q4rsd" Oct 30 20:24:13.562453 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:13.562419 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-64120b/auth-test-model-64120b-kserve-8968677fc-q4rsd" Oct 30 20:24:13.950078 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:13.950055 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-64120b/auth-test-model-64120b-kserve-8968677fc-q4rsd"] Oct 30 20:24:13.952656 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:24:13.952626 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f9c4f3a_c93c_4315_b0c0_95131a3413b6.slice/crio-4b00ab9b7da80444d09f39b560ad0a82661e3990584c5c68e41a941b6ac26ff3 WatchSource:0}: Error finding container 4b00ab9b7da80444d09f39b560ad0a82661e3990584c5c68e41a941b6ac26ff3: Status 404 returned error can't find the container with id 4b00ab9b7da80444d09f39b560ad0a82661e3990584c5c68e41a941b6ac26ff3 Oct 30 20:24:14.433441 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:14.433407 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-64120b/auth-test-model-64120b-kserve-8968677fc-q4rsd" event={"ID":"8f9c4f3a-c93c-4315-b0c0-95131a3413b6","Type":"ContainerStarted","Data":"f2e8ca27df0a0687d642865898b7ee25281a80fae017bc184fc50577dd095cd9"} Oct 30 20:24:14.433842 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:14.433446 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-64120b/auth-test-model-64120b-kserve-8968677fc-q4rsd" event={"ID":"8f9c4f3a-c93c-4315-b0c0-95131a3413b6","Type":"ContainerStarted","Data":"4b00ab9b7da80444d09f39b560ad0a82661e3990584c5c68e41a941b6ac26ff3"} Oct 30 20:24:14.554675 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:14.554615 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-64120b/auth-test-model-64120b-kserve-8968677fc-q4rsd" podStartSLOduration=1.554586688 podStartE2EDuration="1.554586688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:24:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:24:14.553001066 +0000 UTC m=+3558.071316543" watchObservedRunningTime="2026-10-30 20:24:14.554586688 +0000 UTC m=+3558.072902151" Oct 30 20:24:14.563594 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:14.563572 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-64120b/auth-test-model-64120b-kserve-8968677fc-q4rsd" Oct 30 20:24:14.582410 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:14.582383 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ea3800f-f814-4036-b3f4-b7e3573c011b" path="/var/lib/kubelet/pods/4ea3800f-f814-4036-b3f4-b7e3573c011b/volumes" Oct 30 20:24:14.582876 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:14.582862 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de3476ef-83a0-4ca2-9b76-227ee4b35d7c" path="/var/lib/kubelet/pods/de3476ef-83a0-4ca2-9b76-227ee4b35d7c/volumes" Oct 30 20:24:23.083110 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:23.083068 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4b0bf3-mjmdc"] Oct 30 20:24:23.133283 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:23.133254 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4b0bf3-mjmdc"] Oct 30 20:24:23.563228 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:23.563195 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-64120b/auth-test-model-64120b-kserve-8968677fc-q4rsd" Oct 30 20:24:23.576212 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:23.576180 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-64120b/auth-test-model-64120b-kserve-8968677fc-q4rsd" Oct 30 20:24:24.509429 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:24.509403 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-64120b/auth-test-model-64120b-kserve-8968677fc-q4rsd" Oct 30 20:24:24.582366 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:24.582326 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f20d10be-ea2a-48ec-a877-64ffb303f56d" path="/var/lib/kubelet/pods/f20d10be-ea2a-48ec-a877-64ffb303f56d/volumes" Oct 30 20:24:27.060656 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:27.060620 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87e7be8d-6ghql"] Oct 30 20:24:27.110215 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:27.110183 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87e7be8d-6ghql"] Oct 30 20:24:28.582439 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:28.582406 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fca60461-40c2-422d-8b41-a72d876b7227" path="/var/lib/kubelet/pods/fca60461-40c2-422d-8b41-a72d876b7227/volumes" Oct 30 20:24:30.592514 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:30.592481 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm"] Oct 30 20:24:30.592993 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:30.592722 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm" podUID="95bc24b1-3be0-49e7-85e8-f90989e4b10b" containerName="istio-proxy" containerID="cri-o://27c4179de42f1ca7dd5df3a2c9a617775181b6430b91376358a38b379f8f7810" gracePeriod=30 Oct 30 20:24:35.872358 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:35.872289 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm" Oct 30 20:24:35.902490 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:35.902453 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/95bc24b1-3be0-49e7-85e8-f90989e4b10b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95bc24b1-3be0-49e7-85e8-f90989e4b10b-istio-podinfo\") pod \"95bc24b1-3be0-49e7-85e8-f90989e4b10b\" (UID: \"95bc24b1-3be0-49e7-85e8-f90989e4b10b\") " Oct 30 20:24:35.902665 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:35.902530 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95bc24b1-3be0-49e7-85e8-f90989e4b10b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95bc24b1-3be0-49e7-85e8-f90989e4b10b-workload-certs\") pod \"95bc24b1-3be0-49e7-85e8-f90989e4b10b\" (UID: \"95bc24b1-3be0-49e7-85e8-f90989e4b10b\") " Oct 30 20:24:35.902665 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:35.902573 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95bc24b1-3be0-49e7-85e8-f90989e4b10b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95bc24b1-3be0-49e7-85e8-f90989e4b10b-istio-envoy\") pod \"95bc24b1-3be0-49e7-85e8-f90989e4b10b\" (UID: \"95bc24b1-3be0-49e7-85e8-f90989e4b10b\") " Oct 30 20:24:35.902665 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:35.902604 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95bc24b1-3be0-49e7-85e8-f90989e4b10b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95bc24b1-3be0-49e7-85e8-f90989e4b10b-workload-socket\") pod \"95bc24b1-3be0-49e7-85e8-f90989e4b10b\" (UID: \"95bc24b1-3be0-49e7-85e8-f90989e4b10b\") " Oct 30 20:24:35.902825 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:35.902663 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95bc24b1-3be0-49e7-85e8-f90989e4b10b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95bc24b1-3be0-49e7-85e8-f90989e4b10b-istiod-ca-cert\") pod \"95bc24b1-3be0-49e7-85e8-f90989e4b10b\" (UID: \"95bc24b1-3be0-49e7-85e8-f90989e4b10b\") " Oct 30 20:24:35.902825 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:35.902691 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95bc24b1-3be0-49e7-85e8-f90989e4b10b-istio-token\" (UniqueName: \"kubernetes.io/projected/95bc24b1-3be0-49e7-85e8-f90989e4b10b-istio-token\") pod \"95bc24b1-3be0-49e7-85e8-f90989e4b10b\" (UID: \"95bc24b1-3be0-49e7-85e8-f90989e4b10b\") " Oct 30 20:24:35.902825 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:35.902755 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95bc24b1-3be0-49e7-85e8-f90989e4b10b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95bc24b1-3be0-49e7-85e8-f90989e4b10b-credential-socket\") pod \"95bc24b1-3be0-49e7-85e8-f90989e4b10b\" (UID: \"95bc24b1-3be0-49e7-85e8-f90989e4b10b\") " Oct 30 20:24:35.902825 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:35.902815 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95bc24b1-3be0-49e7-85e8-f90989e4b10b-kube-api-access-x2jb9\" (UniqueName: \"kubernetes.io/projected/95bc24b1-3be0-49e7-85e8-f90989e4b10b-kube-api-access-x2jb9\") pod \"95bc24b1-3be0-49e7-85e8-f90989e4b10b\" (UID: \"95bc24b1-3be0-49e7-85e8-f90989e4b10b\") " Oct 30 20:24:35.903043 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:35.902906 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95bc24b1-3be0-49e7-85e8-f90989e4b10b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95bc24b1-3be0-49e7-85e8-f90989e4b10b-istio-data\") pod \"95bc24b1-3be0-49e7-85e8-f90989e4b10b\" (UID: \"95bc24b1-3be0-49e7-85e8-f90989e4b10b\") " Oct 30 20:24:35.903043 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:35.902812 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95bc24b1-3be0-49e7-85e8-f90989e4b10b-workload-certs" pod "95bc24b1-3be0-49e7-85e8-f90989e4b10b" (UID: "95bc24b1-3be0-49e7-85e8-f90989e4b10b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:24:35.903043 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:35.903012 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95bc24b1-3be0-49e7-85e8-f90989e4b10b-credential-socket" pod "95bc24b1-3be0-49e7-85e8-f90989e4b10b" (UID: "95bc24b1-3be0-49e7-85e8-f90989e4b10b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:24:35.903043 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:35.902986 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95bc24b1-3be0-49e7-85e8-f90989e4b10b-workload-socket" pod "95bc24b1-3be0-49e7-85e8-f90989e4b10b" (UID: "95bc24b1-3be0-49e7-85e8-f90989e4b10b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:24:35.903335 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:35.903288 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95bc24b1-3be0-49e7-85e8-f90989e4b10b-istio-data" pod "95bc24b1-3be0-49e7-85e8-f90989e4b10b" (UID: "95bc24b1-3be0-49e7-85e8-f90989e4b10b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:24:35.903426 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:35.903367 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95bc24b1-3be0-49e7-85e8-f90989e4b10b-workload-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:24:35.903426 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:35.903399 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95bc24b1-3be0-49e7-85e8-f90989e4b10b-workload-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:24:35.903426 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:35.903416 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95bc24b1-3be0-49e7-85e8-f90989e4b10b-credential-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:24:35.904590 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:35.904560 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95bc24b1-3be0-49e7-85e8-f90989e4b10b-istiod-ca-cert" pod "95bc24b1-3be0-49e7-85e8-f90989e4b10b" (UID: "95bc24b1-3be0-49e7-85e8-f90989e4b10b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:24:35.905277 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:35.905249 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95bc24b1-3be0-49e7-85e8-f90989e4b10b-istio-envoy" pod "95bc24b1-3be0-49e7-85e8-f90989e4b10b" (UID: "95bc24b1-3be0-49e7-85e8-f90989e4b10b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:24:35.908391 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:35.908327 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/95bc24b1-3be0-49e7-85e8-f90989e4b10b-istio-podinfo" pod "95bc24b1-3be0-49e7-85e8-f90989e4b10b" (UID: "95bc24b1-3be0-49e7-85e8-f90989e4b10b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:24:35.908481 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:35.908459 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95bc24b1-3be0-49e7-85e8-f90989e4b10b-kube-api-access-x2jb9" pod "95bc24b1-3be0-49e7-85e8-f90989e4b10b" (UID: "95bc24b1-3be0-49e7-85e8-f90989e4b10b"). InnerVolumeSpecName "kube-api-access-x2jb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:24:35.909077 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:35.909058 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95bc24b1-3be0-49e7-85e8-f90989e4b10b-istio-token" pod "95bc24b1-3be0-49e7-85e8-f90989e4b10b" (UID: "95bc24b1-3be0-49e7-85e8-f90989e4b10b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:24:36.004401 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:36.004343 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95bc24b1-3be0-49e7-85e8-f90989e4b10b-istio-envoy\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:24:36.004401 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:36.004395 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95bc24b1-3be0-49e7-85e8-f90989e4b10b-istiod-ca-cert\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:24:36.004401 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:36.004406 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95bc24b1-3be0-49e7-85e8-f90989e4b10b-istio-token\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:24:36.004641 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:36.004419 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2jb9\" (UniqueName: \"kubernetes.io/projected/95bc24b1-3be0-49e7-85e8-f90989e4b10b-kube-api-access-x2jb9\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:24:36.004641 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:36.004428 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95bc24b1-3be0-49e7-85e8-f90989e4b10b-istio-data\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:24:36.004641 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:36.004437 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95bc24b1-3be0-49e7-85e8-f90989e4b10b-istio-podinfo\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:24:36.572270 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:36.572239 4292 generic.go:356] "Generic (PLEG): container finished" podID="95bc24b1-3be0-49e7-85e8-f90989e4b10b" containerID="27c4179de42f1ca7dd5df3a2c9a617775181b6430b91376358a38b379f8f7810" exitCode=0 Oct 30 20:24:36.572460 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:36.572307 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm" Oct 30 20:24:36.572460 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:36.572316 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm" event={"ID":"95bc24b1-3be0-49e7-85e8-f90989e4b10b","Type":"ContainerDied","Data":"27c4179de42f1ca7dd5df3a2c9a617775181b6430b91376358a38b379f8f7810"} Oct 30 20:24:36.572460 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:36.572407 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm" event={"ID":"95bc24b1-3be0-49e7-85e8-f90989e4b10b","Type":"ContainerDied","Data":"71ba824349bd05b6ded1d0d1697b884ba64c497ef95f993705bc7cd6f4a96df0"} Oct 30 20:24:36.572460 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:36.572433 4292 scope.go:122] "RemoveContainer" containerID="27c4179de42f1ca7dd5df3a2c9a617775181b6430b91376358a38b379f8f7810" Oct 30 20:24:36.588202 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:36.588182 4292 scope.go:122] "RemoveContainer" containerID="5322434befe396b056f2096ee7d9620699cf1bd709ee1e7aedb10655cacbb465" Oct 30 20:24:36.598127 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:36.598109 4292 scope.go:122] "RemoveContainer" containerID="27c4179de42f1ca7dd5df3a2c9a617775181b6430b91376358a38b379f8f7810" Oct 30 20:24:36.598458 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:24:36.598439 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27c4179de42f1ca7dd5df3a2c9a617775181b6430b91376358a38b379f8f7810\": container with ID starting with 27c4179de42f1ca7dd5df3a2c9a617775181b6430b91376358a38b379f8f7810 not found: ID does not exist" containerID="27c4179de42f1ca7dd5df3a2c9a617775181b6430b91376358a38b379f8f7810" Oct 30 20:24:36.598505 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:36.598469 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27c4179de42f1ca7dd5df3a2c9a617775181b6430b91376358a38b379f8f7810"} err="failed to get container status \"27c4179de42f1ca7dd5df3a2c9a617775181b6430b91376358a38b379f8f7810\": rpc error: code = NotFound desc = could not find container \"27c4179de42f1ca7dd5df3a2c9a617775181b6430b91376358a38b379f8f7810\": container with ID starting with 27c4179de42f1ca7dd5df3a2c9a617775181b6430b91376358a38b379f8f7810 not found: ID does not exist" Oct 30 20:24:36.598505 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:36.598490 4292 scope.go:122] "RemoveContainer" containerID="5322434befe396b056f2096ee7d9620699cf1bd709ee1e7aedb10655cacbb465" Oct 30 20:24:36.598775 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:24:36.598755 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5322434befe396b056f2096ee7d9620699cf1bd709ee1e7aedb10655cacbb465\": container with ID starting with 5322434befe396b056f2096ee7d9620699cf1bd709ee1e7aedb10655cacbb465 not found: ID does not exist" containerID="5322434befe396b056f2096ee7d9620699cf1bd709ee1e7aedb10655cacbb465" Oct 30 20:24:36.598820 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:36.598783 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5322434befe396b056f2096ee7d9620699cf1bd709ee1e7aedb10655cacbb465"} err="failed to get container status \"5322434befe396b056f2096ee7d9620699cf1bd709ee1e7aedb10655cacbb465\": rpc error: code = NotFound desc = could not find container \"5322434befe396b056f2096ee7d9620699cf1bd709ee1e7aedb10655cacbb465\": container with ID starting with 5322434befe396b056f2096ee7d9620699cf1bd709ee1e7aedb10655cacbb465 not found: ID does not exist" Oct 30 20:24:36.746221 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:36.746185 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm"] Oct 30 20:24:36.797019 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:36.796983 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-4b0bf3-openshift-default-775655bcd9-jzptm"] Oct 30 20:24:38.583027 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:38.582996 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95bc24b1-3be0-49e7-85e8-f90989e4b10b" path="/var/lib/kubelet/pods/95bc24b1-3be0-49e7-85e8-f90989e4b10b/volumes" Oct 30 20:24:43.221275 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:24:43.221239 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:24:57.812765 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:24:57.812734 4292 scope.go:122] "RemoveContainer" containerID="83920ddaa203d81fab042b15129afc8c7fa80670f7a103156d9b984d3123ea99" Oct 30 20:25:13.220468 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:25:13.220426 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:25:27.031758 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:25:27.031723 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-bd4c7f/noten-model-bd4c7f-kserve-86b9c9d45c-vbfg6"] Oct 30 20:25:27.032380 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:25:27.032328 4292 cpu_manager.go:414] "containerMap: removing container" podUID="95bc24b1-3be0-49e7-85e8-f90989e4b10b" containerName="istio-proxy" Oct 30 20:25:27.032380 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:25:27.032366 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95bc24b1-3be0-49e7-85e8-f90989e4b10b" containerName="istio-proxy" Oct 30 20:25:27.032554 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:25:27.032544 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="95bc24b1-3be0-49e7-85e8-f90989e4b10b" containerName="istio-proxy" Oct 30 20:25:27.032609 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:25:27.032560 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="95bc24b1-3be0-49e7-85e8-f90989e4b10b" containerName="istio-proxy" Oct 30 20:25:27.035758 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:25:27.035738 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-bd4c7f/noten-model-bd4c7f-kserve-86b9c9d45c-vbfg6" Oct 30 20:25:27.090434 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:25:27.090402 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-bd4c7f/noten-model-bd4c7f-kserve-86b9c9d45c-vbfg6"] Oct 30 20:25:27.106112 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:25:27.106084 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e83370ae-6ade-4183-aaeb-5db5f3598b9f-model-cache\") pod \"noten-model-bd4c7f-kserve-86b9c9d45c-vbfg6\" (UID: \"e83370ae-6ade-4183-aaeb-5db5f3598b9f\") " pod="ai-tenant-e2e-noten-bd4c7f/noten-model-bd4c7f-kserve-86b9c9d45c-vbfg6" Oct 30 20:25:27.106257 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:25:27.106127 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e83370ae-6ade-4183-aaeb-5db5f3598b9f-tmp-dir\") pod \"noten-model-bd4c7f-kserve-86b9c9d45c-vbfg6\" (UID: \"e83370ae-6ade-4183-aaeb-5db5f3598b9f\") " pod="ai-tenant-e2e-noten-bd4c7f/noten-model-bd4c7f-kserve-86b9c9d45c-vbfg6" Oct 30 20:25:27.106257 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:25:27.106245 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrvgt\" (UniqueName: \"kubernetes.io/projected/e83370ae-6ade-4183-aaeb-5db5f3598b9f-kube-api-access-vrvgt\") pod \"noten-model-bd4c7f-kserve-86b9c9d45c-vbfg6\" (UID: \"e83370ae-6ade-4183-aaeb-5db5f3598b9f\") " pod="ai-tenant-e2e-noten-bd4c7f/noten-model-bd4c7f-kserve-86b9c9d45c-vbfg6" Oct 30 20:25:27.106386 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:25:27.106366 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e83370ae-6ade-4183-aaeb-5db5f3598b9f-home\") pod \"noten-model-bd4c7f-kserve-86b9c9d45c-vbfg6\" (UID: \"e83370ae-6ade-4183-aaeb-5db5f3598b9f\") " pod="ai-tenant-e2e-noten-bd4c7f/noten-model-bd4c7f-kserve-86b9c9d45c-vbfg6" Oct 30 20:25:27.106429 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:25:27.106418 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e83370ae-6ade-4183-aaeb-5db5f3598b9f-dshm\") pod \"noten-model-bd4c7f-kserve-86b9c9d45c-vbfg6\" (UID: \"e83370ae-6ade-4183-aaeb-5db5f3598b9f\") " pod="ai-tenant-e2e-noten-bd4c7f/noten-model-bd4c7f-kserve-86b9c9d45c-vbfg6" Oct 30 20:25:27.106490 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:25:27.106436 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e83370ae-6ade-4183-aaeb-5db5f3598b9f-tls-certs\") pod \"noten-model-bd4c7f-kserve-86b9c9d45c-vbfg6\" (UID: \"e83370ae-6ade-4183-aaeb-5db5f3598b9f\") " pod="ai-tenant-e2e-noten-bd4c7f/noten-model-bd4c7f-kserve-86b9c9d45c-vbfg6" Oct 30 20:25:27.138646 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:25:27.138613 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-bd4c7f\"/\"noten-model-bd4c7f-kserve-self-signed-certs\"" Oct 30 20:25:27.138646 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:25:27.138636 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-bd4c7f\"/\"default-dockercfg-592p8\"" Oct 30 20:25:27.138865 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:25:27.138754 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-bd4c7f\"/\"kube-root-ca.crt\"" Oct 30 20:25:27.138865 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:25:27.138830 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-bd4c7f\"/\"openshift-service-ca.crt\"" Oct 30 20:25:27.209069 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:25:27.209026 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e83370ae-6ade-4183-aaeb-5db5f3598b9f-model-cache\") pod \"noten-model-bd4c7f-kserve-86b9c9d45c-vbfg6\" (UID: \"e83370ae-6ade-4183-aaeb-5db5f3598b9f\") " pod="ai-tenant-e2e-noten-bd4c7f/noten-model-bd4c7f-kserve-86b9c9d45c-vbfg6" Oct 30 20:25:27.209300 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:25:27.209090 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e83370ae-6ade-4183-aaeb-5db5f3598b9f-tmp-dir\") pod \"noten-model-bd4c7f-kserve-86b9c9d45c-vbfg6\" (UID: \"e83370ae-6ade-4183-aaeb-5db5f3598b9f\") " pod="ai-tenant-e2e-noten-bd4c7f/noten-model-bd4c7f-kserve-86b9c9d45c-vbfg6" Oct 30 20:25:27.209300 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:25:27.209152 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrvgt\" (UniqueName: \"kubernetes.io/projected/e83370ae-6ade-4183-aaeb-5db5f3598b9f-kube-api-access-vrvgt\") pod \"noten-model-bd4c7f-kserve-86b9c9d45c-vbfg6\" (UID: \"e83370ae-6ade-4183-aaeb-5db5f3598b9f\") " pod="ai-tenant-e2e-noten-bd4c7f/noten-model-bd4c7f-kserve-86b9c9d45c-vbfg6" Oct 30 20:25:27.209300 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:25:27.209234 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e83370ae-6ade-4183-aaeb-5db5f3598b9f-home\") pod \"noten-model-bd4c7f-kserve-86b9c9d45c-vbfg6\" (UID: \"e83370ae-6ade-4183-aaeb-5db5f3598b9f\") " pod="ai-tenant-e2e-noten-bd4c7f/noten-model-bd4c7f-kserve-86b9c9d45c-vbfg6" Oct 30 20:25:27.209300 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:25:27.209287 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e83370ae-6ade-4183-aaeb-5db5f3598b9f-dshm\") pod \"noten-model-bd4c7f-kserve-86b9c9d45c-vbfg6\" (UID: \"e83370ae-6ade-4183-aaeb-5db5f3598b9f\") " pod="ai-tenant-e2e-noten-bd4c7f/noten-model-bd4c7f-kserve-86b9c9d45c-vbfg6" Oct 30 20:25:27.209564 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:25:27.209312 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e83370ae-6ade-4183-aaeb-5db5f3598b9f-tls-certs\") pod \"noten-model-bd4c7f-kserve-86b9c9d45c-vbfg6\" (UID: \"e83370ae-6ade-4183-aaeb-5db5f3598b9f\") " pod="ai-tenant-e2e-noten-bd4c7f/noten-model-bd4c7f-kserve-86b9c9d45c-vbfg6" Oct 30 20:25:27.210283 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:25:27.210253 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e83370ae-6ade-4183-aaeb-5db5f3598b9f-model-cache\") pod \"noten-model-bd4c7f-kserve-86b9c9d45c-vbfg6\" (UID: \"e83370ae-6ade-4183-aaeb-5db5f3598b9f\") " pod="ai-tenant-e2e-noten-bd4c7f/noten-model-bd4c7f-kserve-86b9c9d45c-vbfg6" Oct 30 20:25:27.210488 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:25:27.210465 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e83370ae-6ade-4183-aaeb-5db5f3598b9f-tmp-dir\") pod \"noten-model-bd4c7f-kserve-86b9c9d45c-vbfg6\" (UID: \"e83370ae-6ade-4183-aaeb-5db5f3598b9f\") " pod="ai-tenant-e2e-noten-bd4c7f/noten-model-bd4c7f-kserve-86b9c9d45c-vbfg6" Oct 30 20:25:27.210903 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:25:27.210884 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e83370ae-6ade-4183-aaeb-5db5f3598b9f-home\") pod \"noten-model-bd4c7f-kserve-86b9c9d45c-vbfg6\" (UID: \"e83370ae-6ade-4183-aaeb-5db5f3598b9f\") " pod="ai-tenant-e2e-noten-bd4c7f/noten-model-bd4c7f-kserve-86b9c9d45c-vbfg6" Oct 30 20:25:27.213024 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:25:27.212994 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e83370ae-6ade-4183-aaeb-5db5f3598b9f-dshm\") pod \"noten-model-bd4c7f-kserve-86b9c9d45c-vbfg6\" (UID: \"e83370ae-6ade-4183-aaeb-5db5f3598b9f\") " pod="ai-tenant-e2e-noten-bd4c7f/noten-model-bd4c7f-kserve-86b9c9d45c-vbfg6" Oct 30 20:25:27.213591 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:25:27.213569 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e83370ae-6ade-4183-aaeb-5db5f3598b9f-tls-certs\") pod \"noten-model-bd4c7f-kserve-86b9c9d45c-vbfg6\" (UID: \"e83370ae-6ade-4183-aaeb-5db5f3598b9f\") " pod="ai-tenant-e2e-noten-bd4c7f/noten-model-bd4c7f-kserve-86b9c9d45c-vbfg6" Oct 30 20:25:27.274702 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:25:27.274671 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrvgt\" (UniqueName: \"kubernetes.io/projected/e83370ae-6ade-4183-aaeb-5db5f3598b9f-kube-api-access-vrvgt\") pod \"noten-model-bd4c7f-kserve-86b9c9d45c-vbfg6\" (UID: \"e83370ae-6ade-4183-aaeb-5db5f3598b9f\") " pod="ai-tenant-e2e-noten-bd4c7f/noten-model-bd4c7f-kserve-86b9c9d45c-vbfg6" Oct 30 20:25:27.347541 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:25:27.347465 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-bd4c7f/noten-model-bd4c7f-kserve-86b9c9d45c-vbfg6" Oct 30 20:25:27.738493 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:25:27.738464 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-bd4c7f/noten-model-bd4c7f-kserve-86b9c9d45c-vbfg6"] Oct 30 20:25:27.740140 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:25:27.740114 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode83370ae_6ade_4183_aaeb_5db5f3598b9f.slice/crio-97db54edd04be0caa66e060d40e7d547e2787769230f48bdbc33c2ce04e9e923 WatchSource:0}: Error finding container 97db54edd04be0caa66e060d40e7d547e2787769230f48bdbc33c2ce04e9e923: Status 404 returned error can't find the container with id 97db54edd04be0caa66e060d40e7d547e2787769230f48bdbc33c2ce04e9e923 Oct 30 20:25:27.854422 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:25:27.854384 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-bd4c7f/noten-model-bd4c7f-kserve-86b9c9d45c-vbfg6" event={"ID":"e83370ae-6ade-4183-aaeb-5db5f3598b9f","Type":"ContainerStarted","Data":"0f3ca54146ebcf0445644afb5227a2a63f6aa921e6ae91dfd97708c3441397b5"} Oct 30 20:25:27.854605 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:25:27.854438 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-bd4c7f/noten-model-bd4c7f-kserve-86b9c9d45c-vbfg6" event={"ID":"e83370ae-6ade-4183-aaeb-5db5f3598b9f","Type":"ContainerStarted","Data":"97db54edd04be0caa66e060d40e7d547e2787769230f48bdbc33c2ce04e9e923"} Oct 30 20:25:27.975313 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:25:27.975265 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-bd4c7f/noten-model-bd4c7f-kserve-86b9c9d45c-vbfg6" podStartSLOduration=1.975248005 podStartE2EDuration="1.975248005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:25:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:25:27.974381429 +0000 UTC m=+3631.492696907" watchObservedRunningTime="2026-10-30 20:25:27.975248005 +0000 UTC m=+3631.493563462" Oct 30 20:25:28.348306 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:25:28.348278 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-bd4c7f/noten-model-bd4c7f-kserve-86b9c9d45c-vbfg6" Oct 30 20:25:37.348035 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:25:37.347997 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-bd4c7f/noten-model-bd4c7f-kserve-86b9c9d45c-vbfg6" Oct 30 20:25:37.360575 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:25:37.360546 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-bd4c7f/noten-model-bd4c7f-kserve-86b9c9d45c-vbfg6" Oct 30 20:25:37.919770 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:25:37.919742 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-bd4c7f/noten-model-bd4c7f-kserve-86b9c9d45c-vbfg6" Oct 30 20:25:43.221169 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:25:43.221082 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:25:51.907921 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:25:51.907887 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-bd4c7f/noten-model-bd4c7f-kserve-86b9c9d45c-vbfg6"] Oct 30 20:25:57.855379 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:25:57.855331 4292 scope.go:122] "RemoveContainer" containerID="d68d51e488232e361ad48b09a7e5b43be7e0c7518d4185590fed5e4192cc00f9" Oct 30 20:26:06.928090 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:06.928039 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-bd4c7f/noten-model-bd4c7f-kserve-86b9c9d45c-vbfg6" podUID="e83370ae-6ade-4183-aaeb-5db5f3598b9f" containerName="main" containerID="cri-o://0f3ca54146ebcf0445644afb5227a2a63f6aa921e6ae91dfd97708c3441397b5" gracePeriod=45 Oct 30 20:26:07.071582 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:07.071549 4292 generic.go:356] "Generic (PLEG): container finished" podID="e83370ae-6ade-4183-aaeb-5db5f3598b9f" containerID="0f3ca54146ebcf0445644afb5227a2a63f6aa921e6ae91dfd97708c3441397b5" exitCode=0 Oct 30 20:26:07.071755 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:07.071628 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-bd4c7f/noten-model-bd4c7f-kserve-86b9c9d45c-vbfg6" event={"ID":"e83370ae-6ade-4183-aaeb-5db5f3598b9f","Type":"ContainerDied","Data":"0f3ca54146ebcf0445644afb5227a2a63f6aa921e6ae91dfd97708c3441397b5"} Oct 30 20:26:07.186155 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:07.186101 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-bd4c7f/noten-model-bd4c7f-kserve-86b9c9d45c-vbfg6" Oct 30 20:26:07.263928 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:07.263902 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e83370ae-6ade-4183-aaeb-5db5f3598b9f-tls-certs\" (UniqueName: \"kubernetes.io/secret/e83370ae-6ade-4183-aaeb-5db5f3598b9f-tls-certs\") pod \"e83370ae-6ade-4183-aaeb-5db5f3598b9f\" (UID: \"e83370ae-6ade-4183-aaeb-5db5f3598b9f\") " Oct 30 20:26:07.264086 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:07.263945 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e83370ae-6ade-4183-aaeb-5db5f3598b9f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e83370ae-6ade-4183-aaeb-5db5f3598b9f-model-cache\") pod \"e83370ae-6ade-4183-aaeb-5db5f3598b9f\" (UID: \"e83370ae-6ade-4183-aaeb-5db5f3598b9f\") " Oct 30 20:26:07.264086 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:07.263989 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e83370ae-6ade-4183-aaeb-5db5f3598b9f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e83370ae-6ade-4183-aaeb-5db5f3598b9f-dshm\") pod \"e83370ae-6ade-4183-aaeb-5db5f3598b9f\" (UID: \"e83370ae-6ade-4183-aaeb-5db5f3598b9f\") " Oct 30 20:26:07.264086 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:07.264056 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e83370ae-6ade-4183-aaeb-5db5f3598b9f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e83370ae-6ade-4183-aaeb-5db5f3598b9f-tmp-dir\") pod \"e83370ae-6ade-4183-aaeb-5db5f3598b9f\" (UID: \"e83370ae-6ade-4183-aaeb-5db5f3598b9f\") " Oct 30 20:26:07.264246 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:07.264131 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e83370ae-6ade-4183-aaeb-5db5f3598b9f-kube-api-access-vrvgt\" (UniqueName: \"kubernetes.io/projected/e83370ae-6ade-4183-aaeb-5db5f3598b9f-kube-api-access-vrvgt\") pod \"e83370ae-6ade-4183-aaeb-5db5f3598b9f\" (UID: \"e83370ae-6ade-4183-aaeb-5db5f3598b9f\") " Oct 30 20:26:07.264246 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:07.264157 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e83370ae-6ade-4183-aaeb-5db5f3598b9f-home\" (UniqueName: \"kubernetes.io/empty-dir/e83370ae-6ade-4183-aaeb-5db5f3598b9f-home\") pod \"e83370ae-6ade-4183-aaeb-5db5f3598b9f\" (UID: \"e83370ae-6ade-4183-aaeb-5db5f3598b9f\") " Oct 30 20:26:07.264384 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:07.264292 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e83370ae-6ade-4183-aaeb-5db5f3598b9f-model-cache" pod "e83370ae-6ade-4183-aaeb-5db5f3598b9f" (UID: "e83370ae-6ade-4183-aaeb-5db5f3598b9f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:26:07.264455 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:07.264345 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e83370ae-6ade-4183-aaeb-5db5f3598b9f-tmp-dir" pod "e83370ae-6ade-4183-aaeb-5db5f3598b9f" (UID: "e83370ae-6ade-4183-aaeb-5db5f3598b9f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:26:07.264534 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:07.264509 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e83370ae-6ade-4183-aaeb-5db5f3598b9f-home" pod "e83370ae-6ade-4183-aaeb-5db5f3598b9f" (UID: "e83370ae-6ade-4183-aaeb-5db5f3598b9f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:26:07.264754 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:07.264731 4292 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e83370ae-6ade-4183-aaeb-5db5f3598b9f-home\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:26:07.264824 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:07.264759 4292 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e83370ae-6ade-4183-aaeb-5db5f3598b9f-model-cache\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:26:07.264824 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:07.264777 4292 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e83370ae-6ade-4183-aaeb-5db5f3598b9f-tmp-dir\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:26:07.266289 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:07.266260 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e83370ae-6ade-4183-aaeb-5db5f3598b9f-tls-certs" pod "e83370ae-6ade-4183-aaeb-5db5f3598b9f" (UID: "e83370ae-6ade-4183-aaeb-5db5f3598b9f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:26:07.266436 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:07.266303 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e83370ae-6ade-4183-aaeb-5db5f3598b9f-dshm" pod "e83370ae-6ade-4183-aaeb-5db5f3598b9f" (UID: "e83370ae-6ade-4183-aaeb-5db5f3598b9f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:26:07.266436 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:07.266327 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e83370ae-6ade-4183-aaeb-5db5f3598b9f-kube-api-access-vrvgt" pod "e83370ae-6ade-4183-aaeb-5db5f3598b9f" (UID: "e83370ae-6ade-4183-aaeb-5db5f3598b9f"). InnerVolumeSpecName "kube-api-access-vrvgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:26:07.365945 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:07.365914 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrvgt\" (UniqueName: \"kubernetes.io/projected/e83370ae-6ade-4183-aaeb-5db5f3598b9f-kube-api-access-vrvgt\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:26:07.365945 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:07.365945 4292 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e83370ae-6ade-4183-aaeb-5db5f3598b9f-tls-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:26:07.365945 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:07.365954 4292 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e83370ae-6ade-4183-aaeb-5db5f3598b9f-dshm\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:26:08.087928 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:08.087895 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-bd4c7f/noten-model-bd4c7f-kserve-86b9c9d45c-vbfg6" Oct 30 20:26:08.087928 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:08.087910 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-bd4c7f/noten-model-bd4c7f-kserve-86b9c9d45c-vbfg6" event={"ID":"e83370ae-6ade-4183-aaeb-5db5f3598b9f","Type":"ContainerDied","Data":"97db54edd04be0caa66e060d40e7d547e2787769230f48bdbc33c2ce04e9e923"} Oct 30 20:26:08.088500 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:08.087962 4292 scope.go:122] "RemoveContainer" containerID="0f3ca54146ebcf0445644afb5227a2a63f6aa921e6ae91dfd97708c3441397b5" Oct 30 20:26:08.257139 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:08.257099 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-bd4c7f/noten-model-bd4c7f-kserve-86b9c9d45c-vbfg6"] Oct 30 20:26:08.307752 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:08.307715 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-bd4c7f/noten-model-bd4c7f-kserve-86b9c9d45c-vbfg6"] Oct 30 20:26:08.582586 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:08.582558 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e83370ae-6ade-4183-aaeb-5db5f3598b9f" path="/var/lib/kubelet/pods/e83370ae-6ade-4183-aaeb-5db5f3598b9f/volumes" Oct 30 20:26:13.222295 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:26:13.222254 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:26:33.441074 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:33.441042 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn"] Oct 30 20:26:33.443261 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:33.441251 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn" podUID="a4f584f0-f6b6-474d-ba63-967f27c5a8ff" containerName="istio-proxy" containerID="cri-o://6c2d4f9573a7c906f3da701cca3bf5245581487db8f35ad991b1b28cf5e2dd6f" gracePeriod=30 Oct 30 20:26:37.756922 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:37.756885 4292 patch_prober.go:29] interesting pod/e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:26:37.757316 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:37.756953 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn" podUID="a4f584f0-f6b6-474d-ba63-967f27c5a8ff" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:26:38.701286 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:38.701261 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn" Oct 30 20:26:38.816729 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:38.816699 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a4f584f0-f6b6-474d-ba63-967f27c5a8ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4f584f0-f6b6-474d-ba63-967f27c5a8ff-istio-podinfo\") pod \"a4f584f0-f6b6-474d-ba63-967f27c5a8ff\" (UID: \"a4f584f0-f6b6-474d-ba63-967f27c5a8ff\") " Oct 30 20:26:38.817172 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:38.816744 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4f584f0-f6b6-474d-ba63-967f27c5a8ff-istio-token\" (UniqueName: \"kubernetes.io/projected/a4f584f0-f6b6-474d-ba63-967f27c5a8ff-istio-token\") pod \"a4f584f0-f6b6-474d-ba63-967f27c5a8ff\" (UID: \"a4f584f0-f6b6-474d-ba63-967f27c5a8ff\") " Oct 30 20:26:38.817172 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:38.816789 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4f584f0-f6b6-474d-ba63-967f27c5a8ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f584f0-f6b6-474d-ba63-967f27c5a8ff-workload-socket\") pod \"a4f584f0-f6b6-474d-ba63-967f27c5a8ff\" (UID: \"a4f584f0-f6b6-474d-ba63-967f27c5a8ff\") " Oct 30 20:26:38.817172 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:38.816823 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4f584f0-f6b6-474d-ba63-967f27c5a8ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4f584f0-f6b6-474d-ba63-967f27c5a8ff-workload-certs\") pod \"a4f584f0-f6b6-474d-ba63-967f27c5a8ff\" (UID: \"a4f584f0-f6b6-474d-ba63-967f27c5a8ff\") " Oct 30 20:26:38.817172 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:38.816866 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4f584f0-f6b6-474d-ba63-967f27c5a8ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4f584f0-f6b6-474d-ba63-967f27c5a8ff-istio-envoy\") pod \"a4f584f0-f6b6-474d-ba63-967f27c5a8ff\" (UID: \"a4f584f0-f6b6-474d-ba63-967f27c5a8ff\") " Oct 30 20:26:38.817172 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:38.816934 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4f584f0-f6b6-474d-ba63-967f27c5a8ff-kube-api-access-49m8k\" (UniqueName: \"kubernetes.io/projected/a4f584f0-f6b6-474d-ba63-967f27c5a8ff-kube-api-access-49m8k\") pod \"a4f584f0-f6b6-474d-ba63-967f27c5a8ff\" (UID: \"a4f584f0-f6b6-474d-ba63-967f27c5a8ff\") " Oct 30 20:26:38.817172 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:38.816975 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4f584f0-f6b6-474d-ba63-967f27c5a8ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f584f0-f6b6-474d-ba63-967f27c5a8ff-credential-socket\") pod \"a4f584f0-f6b6-474d-ba63-967f27c5a8ff\" (UID: \"a4f584f0-f6b6-474d-ba63-967f27c5a8ff\") " Oct 30 20:26:38.817172 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:38.817022 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4f584f0-f6b6-474d-ba63-967f27c5a8ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4f584f0-f6b6-474d-ba63-967f27c5a8ff-istio-data\") pod \"a4f584f0-f6b6-474d-ba63-967f27c5a8ff\" (UID: \"a4f584f0-f6b6-474d-ba63-967f27c5a8ff\") " Oct 30 20:26:38.817172 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:38.817061 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4f584f0-f6b6-474d-ba63-967f27c5a8ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4f584f0-f6b6-474d-ba63-967f27c5a8ff-istiod-ca-cert\") pod \"a4f584f0-f6b6-474d-ba63-967f27c5a8ff\" (UID: \"a4f584f0-f6b6-474d-ba63-967f27c5a8ff\") " Oct 30 20:26:38.817172 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:38.817147 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4f584f0-f6b6-474d-ba63-967f27c5a8ff-workload-socket" pod "a4f584f0-f6b6-474d-ba63-967f27c5a8ff" (UID: "a4f584f0-f6b6-474d-ba63-967f27c5a8ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:26:38.817659 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:38.817171 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4f584f0-f6b6-474d-ba63-967f27c5a8ff-workload-certs" pod "a4f584f0-f6b6-474d-ba63-967f27c5a8ff" (UID: "a4f584f0-f6b6-474d-ba63-967f27c5a8ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:26:38.817659 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:38.817412 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4f584f0-f6b6-474d-ba63-967f27c5a8ff-credential-socket" pod "a4f584f0-f6b6-474d-ba63-967f27c5a8ff" (UID: "a4f584f0-f6b6-474d-ba63-967f27c5a8ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:26:38.817659 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:38.817423 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4f584f0-f6b6-474d-ba63-967f27c5a8ff-istio-data" pod "a4f584f0-f6b6-474d-ba63-967f27c5a8ff" (UID: "a4f584f0-f6b6-474d-ba63-967f27c5a8ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:26:38.817785 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:38.817692 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f584f0-f6b6-474d-ba63-967f27c5a8ff-workload-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:26:38.817785 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:38.817691 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4f584f0-f6b6-474d-ba63-967f27c5a8ff-istiod-ca-cert" pod "a4f584f0-f6b6-474d-ba63-967f27c5a8ff" (UID: "a4f584f0-f6b6-474d-ba63-967f27c5a8ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:26:38.817785 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:38.817712 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4f584f0-f6b6-474d-ba63-967f27c5a8ff-workload-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:26:38.817785 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:38.817727 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f584f0-f6b6-474d-ba63-967f27c5a8ff-credential-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:26:38.817785 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:38.817739 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4f584f0-f6b6-474d-ba63-967f27c5a8ff-istio-data\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:26:38.819626 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:38.819591 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4f584f0-f6b6-474d-ba63-967f27c5a8ff-istio-envoy" pod "a4f584f0-f6b6-474d-ba63-967f27c5a8ff" (UID: "a4f584f0-f6b6-474d-ba63-967f27c5a8ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:26:38.819626 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:38.819611 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4f584f0-f6b6-474d-ba63-967f27c5a8ff-istio-token" pod "a4f584f0-f6b6-474d-ba63-967f27c5a8ff" (UID: "a4f584f0-f6b6-474d-ba63-967f27c5a8ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:26:38.819626 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:38.819602 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4f584f0-f6b6-474d-ba63-967f27c5a8ff-istio-podinfo" pod "a4f584f0-f6b6-474d-ba63-967f27c5a8ff" (UID: "a4f584f0-f6b6-474d-ba63-967f27c5a8ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:26:38.819794 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:38.819699 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4f584f0-f6b6-474d-ba63-967f27c5a8ff-kube-api-access-49m8k" pod "a4f584f0-f6b6-474d-ba63-967f27c5a8ff" (UID: "a4f584f0-f6b6-474d-ba63-967f27c5a8ff"). InnerVolumeSpecName "kube-api-access-49m8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:26:38.919332 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:38.919295 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49m8k\" (UniqueName: \"kubernetes.io/projected/a4f584f0-f6b6-474d-ba63-967f27c5a8ff-kube-api-access-49m8k\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:26:38.919332 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:38.919326 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4f584f0-f6b6-474d-ba63-967f27c5a8ff-istiod-ca-cert\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:26:38.919332 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:38.919338 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4f584f0-f6b6-474d-ba63-967f27c5a8ff-istio-podinfo\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:26:38.919573 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:38.919346 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4f584f0-f6b6-474d-ba63-967f27c5a8ff-istio-token\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:26:38.919573 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:38.919384 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4f584f0-f6b6-474d-ba63-967f27c5a8ff-istio-envoy\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:26:39.250810 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:39.250721 4292 generic.go:356] "Generic (PLEG): container finished" podID="a4f584f0-f6b6-474d-ba63-967f27c5a8ff" containerID="6c2d4f9573a7c906f3da701cca3bf5245581487db8f35ad991b1b28cf5e2dd6f" exitCode=0 Oct 30 20:26:39.250810 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:39.250770 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn" Oct 30 20:26:39.251024 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:39.250805 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn" event={"ID":"a4f584f0-f6b6-474d-ba63-967f27c5a8ff","Type":"ContainerDied","Data":"6c2d4f9573a7c906f3da701cca3bf5245581487db8f35ad991b1b28cf5e2dd6f"} Oct 30 20:26:39.251024 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:39.250859 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn" event={"ID":"a4f584f0-f6b6-474d-ba63-967f27c5a8ff","Type":"ContainerDied","Data":"ece8675f4402c683bedb92e721752af77e57b6acc837be99efe38185e259eb6d"} Oct 30 20:26:39.251024 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:39.250880 4292 scope.go:122] "RemoveContainer" containerID="6c2d4f9573a7c906f3da701cca3bf5245581487db8f35ad991b1b28cf5e2dd6f" Oct 30 20:26:39.260997 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:39.260982 4292 scope.go:122] "RemoveContainer" containerID="6c2d4f9573a7c906f3da701cca3bf5245581487db8f35ad991b1b28cf5e2dd6f" Oct 30 20:26:39.261303 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:26:39.261283 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c2d4f9573a7c906f3da701cca3bf5245581487db8f35ad991b1b28cf5e2dd6f\": container with ID starting with 6c2d4f9573a7c906f3da701cca3bf5245581487db8f35ad991b1b28cf5e2dd6f not found: ID does not exist" containerID="6c2d4f9573a7c906f3da701cca3bf5245581487db8f35ad991b1b28cf5e2dd6f" Oct 30 20:26:39.261422 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:39.261315 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c2d4f9573a7c906f3da701cca3bf5245581487db8f35ad991b1b28cf5e2dd6f"} err="failed to get container status \"6c2d4f9573a7c906f3da701cca3bf5245581487db8f35ad991b1b28cf5e2dd6f\": rpc error: code = NotFound desc = could not find container \"6c2d4f9573a7c906f3da701cca3bf5245581487db8f35ad991b1b28cf5e2dd6f\": container with ID starting with 6c2d4f9573a7c906f3da701cca3bf5245581487db8f35ad991b1b28cf5e2dd6f not found: ID does not exist" Oct 30 20:26:39.423706 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:39.423665 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn"] Oct 30 20:26:39.474513 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:39.474481 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-27msn"] Oct 30 20:26:40.356724 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:40.356688 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl"] Oct 30 20:26:40.357171 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:40.357156 4292 cpu_manager.go:414] "containerMap: removing container" podUID="e83370ae-6ade-4183-aaeb-5db5f3598b9f" containerName="main" Oct 30 20:26:40.357222 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:40.357172 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e83370ae-6ade-4183-aaeb-5db5f3598b9f" containerName="main" Oct 30 20:26:40.357222 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:40.357204 4292 cpu_manager.go:414] "containerMap: removing container" podUID="a4f584f0-f6b6-474d-ba63-967f27c5a8ff" containerName="istio-proxy" Oct 30 20:26:40.357222 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:40.357210 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4f584f0-f6b6-474d-ba63-967f27c5a8ff" containerName="istio-proxy" Oct 30 20:26:40.357313 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:40.357223 4292 cpu_manager.go:414] "containerMap: removing container" podUID="95bc24b1-3be0-49e7-85e8-f90989e4b10b" containerName="istio-proxy" Oct 30 20:26:40.357313 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:40.357228 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95bc24b1-3be0-49e7-85e8-f90989e4b10b" containerName="istio-proxy" Oct 30 20:26:40.357313 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:40.357301 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4f584f0-f6b6-474d-ba63-967f27c5a8ff" containerName="istio-proxy" Oct 30 20:26:40.357313 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:40.357312 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="e83370ae-6ade-4183-aaeb-5db5f3598b9f" containerName="main" Oct 30 20:26:40.361982 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:40.361958 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl" Oct 30 20:26:40.416374 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:40.416112 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-742ff8-openshift-default-dockercfg-g2szh\"" Oct 30 20:26:40.445408 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:40.437172 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7758bd21-5416-4ab0-8c70-59970dc01209-istio-data\") pod \"e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl\" (UID: \"7758bd21-5416-4ab0-8c70-59970dc01209\") " pod="openshift-ingress/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl" Oct 30 20:26:40.445408 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:40.437246 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7758bd21-5416-4ab0-8c70-59970dc01209-istiod-ca-cert\") pod \"e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl\" (UID: \"7758bd21-5416-4ab0-8c70-59970dc01209\") " pod="openshift-ingress/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl" Oct 30 20:26:40.445408 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:40.437285 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7758bd21-5416-4ab0-8c70-59970dc01209-istio-token\") pod \"e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl\" (UID: \"7758bd21-5416-4ab0-8c70-59970dc01209\") " pod="openshift-ingress/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl" Oct 30 20:26:40.445408 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:40.437321 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7ms2\" (UniqueName: \"kubernetes.io/projected/7758bd21-5416-4ab0-8c70-59970dc01209-kube-api-access-z7ms2\") pod \"e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl\" (UID: \"7758bd21-5416-4ab0-8c70-59970dc01209\") " pod="openshift-ingress/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl" Oct 30 20:26:40.445408 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:40.437392 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7758bd21-5416-4ab0-8c70-59970dc01209-istio-envoy\") pod \"e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl\" (UID: \"7758bd21-5416-4ab0-8c70-59970dc01209\") " pod="openshift-ingress/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl" Oct 30 20:26:40.445408 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:40.437469 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7758bd21-5416-4ab0-8c70-59970dc01209-workload-certs\") pod \"e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl\" (UID: \"7758bd21-5416-4ab0-8c70-59970dc01209\") " pod="openshift-ingress/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl" Oct 30 20:26:40.445408 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:40.437508 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7758bd21-5416-4ab0-8c70-59970dc01209-credential-socket\") pod \"e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl\" (UID: \"7758bd21-5416-4ab0-8c70-59970dc01209\") " pod="openshift-ingress/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl" Oct 30 20:26:40.445408 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:40.437560 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7758bd21-5416-4ab0-8c70-59970dc01209-istio-podinfo\") pod \"e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl\" (UID: \"7758bd21-5416-4ab0-8c70-59970dc01209\") " pod="openshift-ingress/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl" Oct 30 20:26:40.445408 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:40.437641 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7758bd21-5416-4ab0-8c70-59970dc01209-workload-socket\") pod \"e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl\" (UID: \"7758bd21-5416-4ab0-8c70-59970dc01209\") " pod="openshift-ingress/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl" Oct 30 20:26:40.478319 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:40.478284 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl"] Oct 30 20:26:40.543375 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:40.539809 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7758bd21-5416-4ab0-8c70-59970dc01209-workload-socket\") pod \"e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl\" (UID: \"7758bd21-5416-4ab0-8c70-59970dc01209\") " pod="openshift-ingress/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl" Oct 30 20:26:40.543375 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:40.539881 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7758bd21-5416-4ab0-8c70-59970dc01209-istio-data\") pod \"e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl\" (UID: \"7758bd21-5416-4ab0-8c70-59970dc01209\") " pod="openshift-ingress/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl" Oct 30 20:26:40.543375 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:40.539924 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7758bd21-5416-4ab0-8c70-59970dc01209-istiod-ca-cert\") pod \"e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl\" (UID: \"7758bd21-5416-4ab0-8c70-59970dc01209\") " pod="openshift-ingress/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl" Oct 30 20:26:40.543375 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:40.539955 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7758bd21-5416-4ab0-8c70-59970dc01209-istio-token\") pod \"e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl\" (UID: \"7758bd21-5416-4ab0-8c70-59970dc01209\") " pod="openshift-ingress/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl" Oct 30 20:26:40.543375 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:40.539993 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7ms2\" (UniqueName: \"kubernetes.io/projected/7758bd21-5416-4ab0-8c70-59970dc01209-kube-api-access-z7ms2\") pod \"e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl\" (UID: \"7758bd21-5416-4ab0-8c70-59970dc01209\") " pod="openshift-ingress/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl" Oct 30 20:26:40.543375 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:40.540034 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7758bd21-5416-4ab0-8c70-59970dc01209-istio-envoy\") pod \"e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl\" (UID: \"7758bd21-5416-4ab0-8c70-59970dc01209\") " pod="openshift-ingress/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl" Oct 30 20:26:40.543375 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:40.540103 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7758bd21-5416-4ab0-8c70-59970dc01209-workload-certs\") pod \"e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl\" (UID: \"7758bd21-5416-4ab0-8c70-59970dc01209\") " pod="openshift-ingress/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl" Oct 30 20:26:40.543375 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:40.540140 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7758bd21-5416-4ab0-8c70-59970dc01209-credential-socket\") pod \"e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl\" (UID: \"7758bd21-5416-4ab0-8c70-59970dc01209\") " pod="openshift-ingress/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl" Oct 30 20:26:40.543375 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:40.540193 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7758bd21-5416-4ab0-8c70-59970dc01209-istio-podinfo\") pod \"e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl\" (UID: \"7758bd21-5416-4ab0-8c70-59970dc01209\") " pod="openshift-ingress/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl" Oct 30 20:26:40.548568 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:40.547707 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7758bd21-5416-4ab0-8c70-59970dc01209-istio-podinfo\") pod \"e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl\" (UID: \"7758bd21-5416-4ab0-8c70-59970dc01209\") " pod="openshift-ingress/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl" Oct 30 20:26:40.549683 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:40.549653 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7758bd21-5416-4ab0-8c70-59970dc01209-workload-certs\") pod \"e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl\" (UID: \"7758bd21-5416-4ab0-8c70-59970dc01209\") " pod="openshift-ingress/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl" Oct 30 20:26:40.551045 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:40.550059 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7758bd21-5416-4ab0-8c70-59970dc01209-workload-socket\") pod \"e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl\" (UID: \"7758bd21-5416-4ab0-8c70-59970dc01209\") " pod="openshift-ingress/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl" Oct 30 20:26:40.551045 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:40.550515 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7758bd21-5416-4ab0-8c70-59970dc01209-istiod-ca-cert\") pod \"e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl\" (UID: \"7758bd21-5416-4ab0-8c70-59970dc01209\") " pod="openshift-ingress/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl" Oct 30 20:26:40.551431 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:40.551329 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7758bd21-5416-4ab0-8c70-59970dc01209-credential-socket\") pod \"e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl\" (UID: \"7758bd21-5416-4ab0-8c70-59970dc01209\") " pod="openshift-ingress/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl" Oct 30 20:26:40.551878 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:40.551479 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7758bd21-5416-4ab0-8c70-59970dc01209-istio-envoy\") pod \"e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl\" (UID: \"7758bd21-5416-4ab0-8c70-59970dc01209\") " pod="openshift-ingress/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl" Oct 30 20:26:40.552064 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:40.552013 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7758bd21-5416-4ab0-8c70-59970dc01209-istio-data\") pod \"e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl\" (UID: \"7758bd21-5416-4ab0-8c70-59970dc01209\") " pod="openshift-ingress/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl" Oct 30 20:26:40.582596 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:40.582564 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4f584f0-f6b6-474d-ba63-967f27c5a8ff" path="/var/lib/kubelet/pods/a4f584f0-f6b6-474d-ba63-967f27c5a8ff/volumes" Oct 30 20:26:40.616867 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:40.616793 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7ms2\" (UniqueName: \"kubernetes.io/projected/7758bd21-5416-4ab0-8c70-59970dc01209-kube-api-access-z7ms2\") pod \"e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl\" (UID: \"7758bd21-5416-4ab0-8c70-59970dc01209\") " pod="openshift-ingress/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl" Oct 30 20:26:40.618136 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:40.618104 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7758bd21-5416-4ab0-8c70-59970dc01209-istio-token\") pod \"e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl\" (UID: \"7758bd21-5416-4ab0-8c70-59970dc01209\") " pod="openshift-ingress/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl" Oct 30 20:26:40.676015 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:40.675982 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl" Oct 30 20:26:41.118736 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:41.118708 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl"] Oct 30 20:26:41.120535 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:26:41.120494 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7758bd21_5416_4ab0_8c70_59970dc01209.slice/crio-5e39370ef99b621ec9ec9bc6205fcea94745bfb4d2a6855d8d24d3cf735d3e37 WatchSource:0}: Error finding container 5e39370ef99b621ec9ec9bc6205fcea94745bfb4d2a6855d8d24d3cf735d3e37: Status 404 returned error can't find the container with id 5e39370ef99b621ec9ec9bc6205fcea94745bfb4d2a6855d8d24d3cf735d3e37 Oct 30 20:26:41.123199 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:41.123163 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:26:41.123290 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:41.123242 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:26:41.123290 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:41.123272 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:26:41.265051 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:41.265014 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl" event={"ID":"7758bd21-5416-4ab0-8c70-59970dc01209","Type":"ContainerStarted","Data":"22d0158a2b2bf2aa55272736c8fc7d1c78739f5d782aa32f1def9a5c0a625dbf"} Oct 30 20:26:41.265247 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:41.265075 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl" event={"ID":"7758bd21-5416-4ab0-8c70-59970dc01209","Type":"ContainerStarted","Data":"5e39370ef99b621ec9ec9bc6205fcea94745bfb4d2a6855d8d24d3cf735d3e37"} Oct 30 20:26:41.449407 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:41.449278 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl" podStartSLOduration=1.449263009 podStartE2EDuration="1.449263009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:26:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:41.449198475 +0000 UTC m=+3704.967513942" watchObservedRunningTime="2026-10-30 20:26:41.449263009 +0000 UTC m=+3704.967578466" Oct 30 20:26:41.676444 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:41.676414 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl" Oct 30 20:26:42.682659 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:42.682625 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl" Oct 30 20:26:43.220378 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:26:43.220325 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:26:43.284326 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:43.284287 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl" Oct 30 20:26:43.285466 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:26:43.285446 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl" Oct 30 20:27:10.169776 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:10.169684 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af833e87-7f5w6"] Oct 30 20:27:10.173704 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:10.173684 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af833e87-7f5w6" Oct 30 20:27:10.230466 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:10.230439 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af833e87-7f5w6"] Oct 30 20:27:10.274818 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:10.274788 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac6c96de-884d-474f-84d7-8c58209986cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af833e87-7f5w6\" (UID: \"ac6c96de-884d-474f-84d7-8c58209986cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af833e87-7f5w6" Oct 30 20:27:10.274969 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:10.274904 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac6c96de-884d-474f-84d7-8c58209986cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af833e87-7f5w6\" (UID: \"ac6c96de-884d-474f-84d7-8c58209986cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af833e87-7f5w6" Oct 30 20:27:10.275023 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:10.275004 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfzc5\" (UniqueName: \"kubernetes.io/projected/ac6c96de-884d-474f-84d7-8c58209986cd-kube-api-access-kfzc5\") pod \"maas-api-revoke-sub-af833e87-7f5w6\" (UID: \"ac6c96de-884d-474f-84d7-8c58209986cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af833e87-7f5w6" Oct 30 20:27:10.277580 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:10.277561 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:27:10.277648 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:10.277576 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\"" Oct 30 20:27:10.376372 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:10.376317 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfzc5\" (UniqueName: \"kubernetes.io/projected/ac6c96de-884d-474f-84d7-8c58209986cd-kube-api-access-kfzc5\") pod \"maas-api-revoke-sub-af833e87-7f5w6\" (UID: \"ac6c96de-884d-474f-84d7-8c58209986cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af833e87-7f5w6" Oct 30 20:27:10.376518 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:10.376400 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac6c96de-884d-474f-84d7-8c58209986cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af833e87-7f5w6\" (UID: \"ac6c96de-884d-474f-84d7-8c58209986cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af833e87-7f5w6" Oct 30 20:27:10.376518 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:10.376461 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac6c96de-884d-474f-84d7-8c58209986cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af833e87-7f5w6\" (UID: \"ac6c96de-884d-474f-84d7-8c58209986cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af833e87-7f5w6" Oct 30 20:27:10.377086 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:10.377064 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac6c96de-884d-474f-84d7-8c58209986cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af833e87-7f5w6\" (UID: \"ac6c96de-884d-474f-84d7-8c58209986cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af833e87-7f5w6" Oct 30 20:27:10.378778 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:10.378756 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac6c96de-884d-474f-84d7-8c58209986cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af833e87-7f5w6\" (UID: \"ac6c96de-884d-474f-84d7-8c58209986cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af833e87-7f5w6" Oct 30 20:27:10.440064 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:10.440002 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfzc5\" (UniqueName: \"kubernetes.io/projected/ac6c96de-884d-474f-84d7-8c58209986cd-kube-api-access-kfzc5\") pod \"maas-api-revoke-sub-af833e87-7f5w6\" (UID: \"ac6c96de-884d-474f-84d7-8c58209986cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af833e87-7f5w6" Oct 30 20:27:10.486345 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:10.486319 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af833e87-7f5w6" Oct 30 20:27:10.870052 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:10.870028 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af833e87-7f5w6"] Oct 30 20:27:10.872049 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:27:10.872019 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac6c96de_884d_474f_84d7_8c58209986cd.slice/crio-08359bdc2d69d329eadc3b1cee041b6f0e92e3a6d3b636261482ba3192c972e1 WatchSource:0}: Error finding container 08359bdc2d69d329eadc3b1cee041b6f0e92e3a6d3b636261482ba3192c972e1: Status 404 returned error can't find the container with id 08359bdc2d69d329eadc3b1cee041b6f0e92e3a6d3b636261482ba3192c972e1 Oct 30 20:27:11.459290 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:11.459202 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af833e87-7f5w6" event={"ID":"ac6c96de-884d-474f-84d7-8c58209986cd","Type":"ContainerStarted","Data":"aa99f136bf5eca6178981da3f088eab91d52360162828584f13b2d9b891960a8"} Oct 30 20:27:11.459290 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:11.459268 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af833e87-7f5w6" event={"ID":"ac6c96de-884d-474f-84d7-8c58209986cd","Type":"ContainerStarted","Data":"08359bdc2d69d329eadc3b1cee041b6f0e92e3a6d3b636261482ba3192c972e1"} Oct 30 20:27:11.578588 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:11.578535 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af833e87-7f5w6" podStartSLOduration=1.578520949 podStartE2EDuration="1.578520949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:27:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:27:11.577516935 +0000 UTC m=+3735.095832393" watchObservedRunningTime="2026-10-30 20:27:11.578520949 +0000 UTC m=+3735.096836405" Oct 30 20:27:12.466036 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:12.465976 4292 generic.go:356] "Generic (PLEG): container finished" podID="ac6c96de-884d-474f-84d7-8c58209986cd" containerID="aa99f136bf5eca6178981da3f088eab91d52360162828584f13b2d9b891960a8" exitCode=0 Oct 30 20:27:12.466448 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:12.466058 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af833e87-7f5w6" event={"ID":"ac6c96de-884d-474f-84d7-8c58209986cd","Type":"ContainerDied","Data":"aa99f136bf5eca6178981da3f088eab91d52360162828584f13b2d9b891960a8"} Oct 30 20:27:13.221184 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:27:13.221151 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:27:13.622188 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:13.622164 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af833e87-7f5w6" Oct 30 20:27:13.711705 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:13.711673 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac6c96de-884d-474f-84d7-8c58209986cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac6c96de-884d-474f-84d7-8c58209986cd-maas-api-xks-ca\") pod \"ac6c96de-884d-474f-84d7-8c58209986cd\" (UID: \"ac6c96de-884d-474f-84d7-8c58209986cd\") " Oct 30 20:27:13.711895 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:13.711787 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac6c96de-884d-474f-84d7-8c58209986cd-kube-api-access-kfzc5\" (UniqueName: \"kubernetes.io/projected/ac6c96de-884d-474f-84d7-8c58209986cd-kube-api-access-kfzc5\") pod \"ac6c96de-884d-474f-84d7-8c58209986cd\" (UID: \"ac6c96de-884d-474f-84d7-8c58209986cd\") " Oct 30 20:27:13.712038 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:13.711943 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac6c96de-884d-474f-84d7-8c58209986cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac6c96de-884d-474f-84d7-8c58209986cd-maas-api-openshift-ca\") pod \"ac6c96de-884d-474f-84d7-8c58209986cd\" (UID: \"ac6c96de-884d-474f-84d7-8c58209986cd\") " Oct 30 20:27:13.712326 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:13.712296 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac6c96de-884d-474f-84d7-8c58209986cd-maas-api-openshift-ca" pod "ac6c96de-884d-474f-84d7-8c58209986cd" (UID: "ac6c96de-884d-474f-84d7-8c58209986cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:27:13.713956 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:13.713924 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac6c96de-884d-474f-84d7-8c58209986cd-maas-api-xks-ca" pod "ac6c96de-884d-474f-84d7-8c58209986cd" (UID: "ac6c96de-884d-474f-84d7-8c58209986cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:27:13.714059 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:13.713967 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac6c96de-884d-474f-84d7-8c58209986cd-kube-api-access-kfzc5" pod "ac6c96de-884d-474f-84d7-8c58209986cd" (UID: "ac6c96de-884d-474f-84d7-8c58209986cd"). InnerVolumeSpecName "kube-api-access-kfzc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:27:13.813859 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:13.813817 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfzc5\" (UniqueName: \"kubernetes.io/projected/ac6c96de-884d-474f-84d7-8c58209986cd-kube-api-access-kfzc5\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:27:13.813859 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:13.813851 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac6c96de-884d-474f-84d7-8c58209986cd-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:27:13.813859 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:13.813866 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac6c96de-884d-474f-84d7-8c58209986cd-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:27:14.478473 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:14.478444 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af833e87-7f5w6" Oct 30 20:27:14.478670 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:14.478483 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af833e87-7f5w6" event={"ID":"ac6c96de-884d-474f-84d7-8c58209986cd","Type":"ContainerDied","Data":"08359bdc2d69d329eadc3b1cee041b6f0e92e3a6d3b636261482ba3192c972e1"} Oct 30 20:27:14.478670 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:14.478562 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08359bdc2d69d329eadc3b1cee041b6f0e92e3a6d3b636261482ba3192c972e1" Oct 30 20:27:20.886629 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:20.886592 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb8de611-lrvmg"] Oct 30 20:27:20.887067 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:20.887052 4292 cpu_manager.go:414] "containerMap: removing container" podUID="ac6c96de-884d-474f-84d7-8c58209986cd" containerName="revoke-keys" Oct 30 20:27:20.887117 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:20.887068 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac6c96de-884d-474f-84d7-8c58209986cd" containerName="revoke-keys" Oct 30 20:27:20.887159 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:20.887150 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac6c96de-884d-474f-84d7-8c58209986cd" containerName="revoke-keys" Oct 30 20:27:20.890611 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:20.890593 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8de611-lrvmg" Oct 30 20:27:20.947556 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:20.947526 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb8de611-lrvmg"] Oct 30 20:27:20.993722 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:20.993691 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/710fb1e6-2030-45fc-91b5-d4f13d6941b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb8de611-lrvmg\" (UID: \"710fb1e6-2030-45fc-91b5-d4f13d6941b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8de611-lrvmg" Oct 30 20:27:20.993881 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:20.993754 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/710fb1e6-2030-45fc-91b5-d4f13d6941b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb8de611-lrvmg\" (UID: \"710fb1e6-2030-45fc-91b5-d4f13d6941b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8de611-lrvmg" Oct 30 20:27:20.993881 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:20.993847 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsrz6\" (UniqueName: \"kubernetes.io/projected/710fb1e6-2030-45fc-91b5-d4f13d6941b9-kube-api-access-lsrz6\") pod \"maas-api-revoke-sub-eb8de611-lrvmg\" (UID: \"710fb1e6-2030-45fc-91b5-d4f13d6941b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8de611-lrvmg" Oct 30 20:27:20.994649 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:20.994624 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:27:20.994781 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:20.994729 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\"" Oct 30 20:27:21.094877 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:21.094847 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsrz6\" (UniqueName: \"kubernetes.io/projected/710fb1e6-2030-45fc-91b5-d4f13d6941b9-kube-api-access-lsrz6\") pod \"maas-api-revoke-sub-eb8de611-lrvmg\" (UID: \"710fb1e6-2030-45fc-91b5-d4f13d6941b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8de611-lrvmg" Oct 30 20:27:21.095029 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:21.094962 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/710fb1e6-2030-45fc-91b5-d4f13d6941b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb8de611-lrvmg\" (UID: \"710fb1e6-2030-45fc-91b5-d4f13d6941b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8de611-lrvmg" Oct 30 20:27:21.095029 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:21.094998 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/710fb1e6-2030-45fc-91b5-d4f13d6941b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb8de611-lrvmg\" (UID: \"710fb1e6-2030-45fc-91b5-d4f13d6941b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8de611-lrvmg" Oct 30 20:27:21.095655 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:21.095632 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/710fb1e6-2030-45fc-91b5-d4f13d6941b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb8de611-lrvmg\" (UID: \"710fb1e6-2030-45fc-91b5-d4f13d6941b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8de611-lrvmg" Oct 30 20:27:21.097425 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:21.097401 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/710fb1e6-2030-45fc-91b5-d4f13d6941b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb8de611-lrvmg\" (UID: \"710fb1e6-2030-45fc-91b5-d4f13d6941b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8de611-lrvmg" Oct 30 20:27:21.158073 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:21.158007 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsrz6\" (UniqueName: \"kubernetes.io/projected/710fb1e6-2030-45fc-91b5-d4f13d6941b9-kube-api-access-lsrz6\") pod \"maas-api-revoke-sub-eb8de611-lrvmg\" (UID: \"710fb1e6-2030-45fc-91b5-d4f13d6941b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8de611-lrvmg" Oct 30 20:27:21.204765 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:21.204732 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8de611-lrvmg" Oct 30 20:27:21.582207 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:21.582185 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb8de611-lrvmg"] Oct 30 20:27:21.585286 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:27:21.585248 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod710fb1e6_2030_45fc_91b5_d4f13d6941b9.slice/crio-bcbd3c1b4319a7279236843e4cb873ebb7e05a4b3a428177225f29ccc9e68614 WatchSource:0}: Error finding container bcbd3c1b4319a7279236843e4cb873ebb7e05a4b3a428177225f29ccc9e68614: Status 404 returned error can't find the container with id bcbd3c1b4319a7279236843e4cb873ebb7e05a4b3a428177225f29ccc9e68614 Oct 30 20:27:22.523082 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:22.523052 4292 generic.go:356] "Generic (PLEG): container finished" podID="710fb1e6-2030-45fc-91b5-d4f13d6941b9" containerID="2a4c87d74b2a295d5a436a0d4bed89d31934944069d4db8567e178cb065793a0" exitCode=0 Oct 30 20:27:22.523520 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:22.523137 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8de611-lrvmg" event={"ID":"710fb1e6-2030-45fc-91b5-d4f13d6941b9","Type":"ContainerDied","Data":"2a4c87d74b2a295d5a436a0d4bed89d31934944069d4db8567e178cb065793a0"} Oct 30 20:27:22.523520 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:22.523201 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8de611-lrvmg" event={"ID":"710fb1e6-2030-45fc-91b5-d4f13d6941b9","Type":"ContainerStarted","Data":"bcbd3c1b4319a7279236843e4cb873ebb7e05a4b3a428177225f29ccc9e68614"} Oct 30 20:27:23.691409 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:23.691383 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8de611-lrvmg" Oct 30 20:27:23.822555 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:23.822473 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/710fb1e6-2030-45fc-91b5-d4f13d6941b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/710fb1e6-2030-45fc-91b5-d4f13d6941b9-maas-api-xks-ca\") pod \"710fb1e6-2030-45fc-91b5-d4f13d6941b9\" (UID: \"710fb1e6-2030-45fc-91b5-d4f13d6941b9\") " Oct 30 20:27:23.822555 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:23.822536 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/710fb1e6-2030-45fc-91b5-d4f13d6941b9-kube-api-access-lsrz6\" (UniqueName: \"kubernetes.io/projected/710fb1e6-2030-45fc-91b5-d4f13d6941b9-kube-api-access-lsrz6\") pod \"710fb1e6-2030-45fc-91b5-d4f13d6941b9\" (UID: \"710fb1e6-2030-45fc-91b5-d4f13d6941b9\") " Oct 30 20:27:23.822792 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:23.822586 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/710fb1e6-2030-45fc-91b5-d4f13d6941b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/710fb1e6-2030-45fc-91b5-d4f13d6941b9-maas-api-openshift-ca\") pod \"710fb1e6-2030-45fc-91b5-d4f13d6941b9\" (UID: \"710fb1e6-2030-45fc-91b5-d4f13d6941b9\") " Oct 30 20:27:23.823077 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:23.823043 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/710fb1e6-2030-45fc-91b5-d4f13d6941b9-maas-api-openshift-ca" pod "710fb1e6-2030-45fc-91b5-d4f13d6941b9" (UID: "710fb1e6-2030-45fc-91b5-d4f13d6941b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:27:23.824795 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:23.824765 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/710fb1e6-2030-45fc-91b5-d4f13d6941b9-maas-api-xks-ca" pod "710fb1e6-2030-45fc-91b5-d4f13d6941b9" (UID: "710fb1e6-2030-45fc-91b5-d4f13d6941b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:27:23.824900 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:23.824804 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/710fb1e6-2030-45fc-91b5-d4f13d6941b9-kube-api-access-lsrz6" pod "710fb1e6-2030-45fc-91b5-d4f13d6941b9" (UID: "710fb1e6-2030-45fc-91b5-d4f13d6941b9"). InnerVolumeSpecName "kube-api-access-lsrz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:27:23.923737 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:23.923705 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/710fb1e6-2030-45fc-91b5-d4f13d6941b9-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:27:23.923737 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:23.923739 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/710fb1e6-2030-45fc-91b5-d4f13d6941b9-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:27:23.923941 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:23.923751 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsrz6\" (UniqueName: \"kubernetes.io/projected/710fb1e6-2030-45fc-91b5-d4f13d6941b9-kube-api-access-lsrz6\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:27:24.536037 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:24.536005 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8de611-lrvmg" event={"ID":"710fb1e6-2030-45fc-91b5-d4f13d6941b9","Type":"ContainerDied","Data":"bcbd3c1b4319a7279236843e4cb873ebb7e05a4b3a428177225f29ccc9e68614"} Oct 30 20:27:24.536037 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:24.536025 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8de611-lrvmg" Oct 30 20:27:24.536243 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:24.536077 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bcbd3c1b4319a7279236843e4cb873ebb7e05a4b3a428177225f29ccc9e68614" Oct 30 20:27:34.522061 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:34.522027 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38554f47-lqw4b"] Oct 30 20:27:34.522543 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:34.522527 4292 cpu_manager.go:414] "containerMap: removing container" podUID="710fb1e6-2030-45fc-91b5-d4f13d6941b9" containerName="revoke-keys" Oct 30 20:27:34.522543 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:34.522544 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="710fb1e6-2030-45fc-91b5-d4f13d6941b9" containerName="revoke-keys" Oct 30 20:27:34.522647 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:34.522638 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="710fb1e6-2030-45fc-91b5-d4f13d6941b9" containerName="revoke-keys" Oct 30 20:27:34.525796 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:34.525780 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38554f47-lqw4b" Oct 30 20:27:34.583634 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:34.583605 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38554f47-lqw4b"] Oct 30 20:27:34.628746 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:34.628717 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\"" Oct 30 20:27:34.628746 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:34.628713 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:27:34.633664 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:34.633634 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr7tp\" (UniqueName: \"kubernetes.io/projected/f18d86a4-efcc-4bb0-91c0-c98e1765acbb-kube-api-access-jr7tp\") pod \"maas-api-revoke-sub-38554f47-lqw4b\" (UID: \"f18d86a4-efcc-4bb0-91c0-c98e1765acbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38554f47-lqw4b" Oct 30 20:27:34.633864 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:34.633842 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f18d86a4-efcc-4bb0-91c0-c98e1765acbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38554f47-lqw4b\" (UID: \"f18d86a4-efcc-4bb0-91c0-c98e1765acbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38554f47-lqw4b" Oct 30 20:27:34.633954 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:34.633906 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f18d86a4-efcc-4bb0-91c0-c98e1765acbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38554f47-lqw4b\" (UID: \"f18d86a4-efcc-4bb0-91c0-c98e1765acbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38554f47-lqw4b" Oct 30 20:27:34.734767 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:34.734736 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr7tp\" (UniqueName: \"kubernetes.io/projected/f18d86a4-efcc-4bb0-91c0-c98e1765acbb-kube-api-access-jr7tp\") pod \"maas-api-revoke-sub-38554f47-lqw4b\" (UID: \"f18d86a4-efcc-4bb0-91c0-c98e1765acbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38554f47-lqw4b" Oct 30 20:27:34.734945 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:34.734784 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f18d86a4-efcc-4bb0-91c0-c98e1765acbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38554f47-lqw4b\" (UID: \"f18d86a4-efcc-4bb0-91c0-c98e1765acbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38554f47-lqw4b" Oct 30 20:27:34.734945 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:34.734816 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f18d86a4-efcc-4bb0-91c0-c98e1765acbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38554f47-lqw4b\" (UID: \"f18d86a4-efcc-4bb0-91c0-c98e1765acbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38554f47-lqw4b" Oct 30 20:27:34.735547 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:34.735526 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f18d86a4-efcc-4bb0-91c0-c98e1765acbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38554f47-lqw4b\" (UID: \"f18d86a4-efcc-4bb0-91c0-c98e1765acbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38554f47-lqw4b" Oct 30 20:27:34.737169 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:34.737149 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f18d86a4-efcc-4bb0-91c0-c98e1765acbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38554f47-lqw4b\" (UID: \"f18d86a4-efcc-4bb0-91c0-c98e1765acbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38554f47-lqw4b" Oct 30 20:27:34.797803 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:34.797744 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr7tp\" (UniqueName: \"kubernetes.io/projected/f18d86a4-efcc-4bb0-91c0-c98e1765acbb-kube-api-access-jr7tp\") pod \"maas-api-revoke-sub-38554f47-lqw4b\" (UID: \"f18d86a4-efcc-4bb0-91c0-c98e1765acbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38554f47-lqw4b" Oct 30 20:27:34.838821 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:34.838787 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38554f47-lqw4b" Oct 30 20:27:35.145886 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:35.145815 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48526fc3-g7k64"] Oct 30 20:27:35.149630 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:35.149612 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48526fc3-g7k64" Oct 30 20:27:35.205072 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:35.205037 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48526fc3-g7k64"] Oct 30 20:27:35.243486 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:35.243449 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f162b4dd-c258-4b49-8612-44b7a83122c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48526fc3-g7k64\" (UID: \"f162b4dd-c258-4b49-8612-44b7a83122c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48526fc3-g7k64" Oct 30 20:27:35.243735 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:35.243714 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb7g2\" (UniqueName: \"kubernetes.io/projected/f162b4dd-c258-4b49-8612-44b7a83122c7-kube-api-access-lb7g2\") pod \"maas-api-revoke-sub-48526fc3-g7k64\" (UID: \"f162b4dd-c258-4b49-8612-44b7a83122c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48526fc3-g7k64" Oct 30 20:27:35.243806 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:35.243762 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f162b4dd-c258-4b49-8612-44b7a83122c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48526fc3-g7k64\" (UID: \"f162b4dd-c258-4b49-8612-44b7a83122c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48526fc3-g7k64" Oct 30 20:27:35.261107 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:35.261012 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38554f47-lqw4b"] Oct 30 20:27:35.344903 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:35.344869 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f162b4dd-c258-4b49-8612-44b7a83122c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48526fc3-g7k64\" (UID: \"f162b4dd-c258-4b49-8612-44b7a83122c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48526fc3-g7k64" Oct 30 20:27:35.344903 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:35.344907 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb7g2\" (UniqueName: \"kubernetes.io/projected/f162b4dd-c258-4b49-8612-44b7a83122c7-kube-api-access-lb7g2\") pod \"maas-api-revoke-sub-48526fc3-g7k64\" (UID: \"f162b4dd-c258-4b49-8612-44b7a83122c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48526fc3-g7k64" Oct 30 20:27:35.345095 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:35.344930 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f162b4dd-c258-4b49-8612-44b7a83122c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48526fc3-g7k64\" (UID: \"f162b4dd-c258-4b49-8612-44b7a83122c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48526fc3-g7k64" Oct 30 20:27:35.345652 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:35.345631 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f162b4dd-c258-4b49-8612-44b7a83122c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48526fc3-g7k64\" (UID: \"f162b4dd-c258-4b49-8612-44b7a83122c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48526fc3-g7k64" Oct 30 20:27:35.347325 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:35.347306 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f162b4dd-c258-4b49-8612-44b7a83122c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48526fc3-g7k64\" (UID: \"f162b4dd-c258-4b49-8612-44b7a83122c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48526fc3-g7k64" Oct 30 20:27:35.407607 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:35.407545 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb7g2\" (UniqueName: \"kubernetes.io/projected/f162b4dd-c258-4b49-8612-44b7a83122c7-kube-api-access-lb7g2\") pod \"maas-api-revoke-sub-48526fc3-g7k64\" (UID: \"f162b4dd-c258-4b49-8612-44b7a83122c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48526fc3-g7k64" Oct 30 20:27:35.463893 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:35.463870 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48526fc3-g7k64" Oct 30 20:27:35.597153 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:35.597122 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38554f47-lqw4b" event={"ID":"f18d86a4-efcc-4bb0-91c0-c98e1765acbb","Type":"ContainerStarted","Data":"8a0fd5add8c6104c2c2771368e0ad9ab985e2cfecfa272d3d285419d2a921593"} Oct 30 20:27:35.597617 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:35.597173 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38554f47-lqw4b" event={"ID":"f18d86a4-efcc-4bb0-91c0-c98e1765acbb","Type":"ContainerStarted","Data":"513a530201a371ddb2b00f4ea0ddb23ccc9f77f3f7c88789fa7543a6ef156b0d"} Oct 30 20:27:35.716324 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:35.716219 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38554f47-lqw4b" podStartSLOduration=1.716198415 podStartE2EDuration="1.716198415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:27:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:27:35.714323335 +0000 UTC m=+3759.232638791" watchObservedRunningTime="2026-10-30 20:27:35.716198415 +0000 UTC m=+3759.234513872" Oct 30 20:27:35.844733 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:35.844707 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48526fc3-g7k64"] Oct 30 20:27:35.846220 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:27:35.846195 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf162b4dd_c258_4b49_8612_44b7a83122c7.slice/crio-40533cbb3bdf485e9a2bf1d45f376385702ab5d227ed796531c1b4e81860d991 WatchSource:0}: Error finding container 40533cbb3bdf485e9a2bf1d45f376385702ab5d227ed796531c1b4e81860d991: Status 404 returned error can't find the container with id 40533cbb3bdf485e9a2bf1d45f376385702ab5d227ed796531c1b4e81860d991 Oct 30 20:27:36.604109 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:36.604067 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48526fc3-g7k64" event={"ID":"f162b4dd-c258-4b49-8612-44b7a83122c7","Type":"ContainerStarted","Data":"f8feed4dcc691609a029f529adc80f22f0fb1c03e9f0324d5b6b2b01cd938d46"} Oct 30 20:27:36.604547 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:36.604127 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48526fc3-g7k64" event={"ID":"f162b4dd-c258-4b49-8612-44b7a83122c7","Type":"ContainerStarted","Data":"40533cbb3bdf485e9a2bf1d45f376385702ab5d227ed796531c1b4e81860d991"} Oct 30 20:27:36.605729 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:36.605708 4292 generic.go:356] "Generic (PLEG): container finished" podID="f18d86a4-efcc-4bb0-91c0-c98e1765acbb" containerID="8a0fd5add8c6104c2c2771368e0ad9ab985e2cfecfa272d3d285419d2a921593" exitCode=0 Oct 30 20:27:36.605787 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:36.605746 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38554f47-lqw4b" event={"ID":"f18d86a4-efcc-4bb0-91c0-c98e1765acbb","Type":"ContainerDied","Data":"8a0fd5add8c6104c2c2771368e0ad9ab985e2cfecfa272d3d285419d2a921593"} Oct 30 20:27:36.725264 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:36.725214 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48526fc3-g7k64" podStartSLOduration=1.7251991819999999 podStartE2EDuration="1.725199182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:27:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:27:36.722845674 +0000 UTC m=+3760.241161132" watchObservedRunningTime="2026-10-30 20:27:36.725199182 +0000 UTC m=+3760.243514638" Oct 30 20:27:37.612983 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:37.612949 4292 generic.go:356] "Generic (PLEG): container finished" podID="f162b4dd-c258-4b49-8612-44b7a83122c7" containerID="f8feed4dcc691609a029f529adc80f22f0fb1c03e9f0324d5b6b2b01cd938d46" exitCode=0 Oct 30 20:27:37.613423 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:37.613020 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48526fc3-g7k64" event={"ID":"f162b4dd-c258-4b49-8612-44b7a83122c7","Type":"ContainerDied","Data":"f8feed4dcc691609a029f529adc80f22f0fb1c03e9f0324d5b6b2b01cd938d46"} Oct 30 20:27:37.756120 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:37.756096 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38554f47-lqw4b" Oct 30 20:27:37.874610 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:37.874509 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f18d86a4-efcc-4bb0-91c0-c98e1765acbb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f18d86a4-efcc-4bb0-91c0-c98e1765acbb-maas-api-openshift-ca\") pod \"f18d86a4-efcc-4bb0-91c0-c98e1765acbb\" (UID: \"f18d86a4-efcc-4bb0-91c0-c98e1765acbb\") " Oct 30 20:27:37.874610 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:37.874604 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f18d86a4-efcc-4bb0-91c0-c98e1765acbb-kube-api-access-jr7tp\" (UniqueName: \"kubernetes.io/projected/f18d86a4-efcc-4bb0-91c0-c98e1765acbb-kube-api-access-jr7tp\") pod \"f18d86a4-efcc-4bb0-91c0-c98e1765acbb\" (UID: \"f18d86a4-efcc-4bb0-91c0-c98e1765acbb\") " Oct 30 20:27:37.875111 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:37.874766 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f18d86a4-efcc-4bb0-91c0-c98e1765acbb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f18d86a4-efcc-4bb0-91c0-c98e1765acbb-maas-api-xks-ca\") pod \"f18d86a4-efcc-4bb0-91c0-c98e1765acbb\" (UID: \"f18d86a4-efcc-4bb0-91c0-c98e1765acbb\") " Oct 30 20:27:37.875111 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:37.874964 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f18d86a4-efcc-4bb0-91c0-c98e1765acbb-maas-api-openshift-ca" pod "f18d86a4-efcc-4bb0-91c0-c98e1765acbb" (UID: "f18d86a4-efcc-4bb0-91c0-c98e1765acbb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:27:37.875233 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:37.875139 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f18d86a4-efcc-4bb0-91c0-c98e1765acbb-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:27:37.877826 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:37.877792 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f18d86a4-efcc-4bb0-91c0-c98e1765acbb-kube-api-access-jr7tp" pod "f18d86a4-efcc-4bb0-91c0-c98e1765acbb" (UID: "f18d86a4-efcc-4bb0-91c0-c98e1765acbb"). InnerVolumeSpecName "kube-api-access-jr7tp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:27:37.881137 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:37.881107 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f18d86a4-efcc-4bb0-91c0-c98e1765acbb-maas-api-xks-ca" pod "f18d86a4-efcc-4bb0-91c0-c98e1765acbb" (UID: "f18d86a4-efcc-4bb0-91c0-c98e1765acbb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:27:37.975847 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:37.975815 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f18d86a4-efcc-4bb0-91c0-c98e1765acbb-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:27:37.975847 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:37.975846 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jr7tp\" (UniqueName: \"kubernetes.io/projected/f18d86a4-efcc-4bb0-91c0-c98e1765acbb-kube-api-access-jr7tp\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:27:38.619951 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:38.619921 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38554f47-lqw4b" Oct 30 20:27:38.620458 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:38.619949 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38554f47-lqw4b" event={"ID":"f18d86a4-efcc-4bb0-91c0-c98e1765acbb","Type":"ContainerDied","Data":"513a530201a371ddb2b00f4ea0ddb23ccc9f77f3f7c88789fa7543a6ef156b0d"} Oct 30 20:27:38.620458 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:38.619994 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="513a530201a371ddb2b00f4ea0ddb23ccc9f77f3f7c88789fa7543a6ef156b0d" Oct 30 20:27:38.759912 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:38.759889 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48526fc3-g7k64" Oct 30 20:27:38.886701 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:38.886620 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f162b4dd-c258-4b49-8612-44b7a83122c7-kube-api-access-lb7g2\" (UniqueName: \"kubernetes.io/projected/f162b4dd-c258-4b49-8612-44b7a83122c7-kube-api-access-lb7g2\") pod \"f162b4dd-c258-4b49-8612-44b7a83122c7\" (UID: \"f162b4dd-c258-4b49-8612-44b7a83122c7\") " Oct 30 20:27:38.886875 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:38.886714 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f162b4dd-c258-4b49-8612-44b7a83122c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f162b4dd-c258-4b49-8612-44b7a83122c7-maas-api-xks-ca\") pod \"f162b4dd-c258-4b49-8612-44b7a83122c7\" (UID: \"f162b4dd-c258-4b49-8612-44b7a83122c7\") " Oct 30 20:27:38.886875 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:38.886823 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f162b4dd-c258-4b49-8612-44b7a83122c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f162b4dd-c258-4b49-8612-44b7a83122c7-maas-api-openshift-ca\") pod \"f162b4dd-c258-4b49-8612-44b7a83122c7\" (UID: \"f162b4dd-c258-4b49-8612-44b7a83122c7\") " Oct 30 20:27:38.887341 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:38.887316 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f162b4dd-c258-4b49-8612-44b7a83122c7-maas-api-openshift-ca" pod "f162b4dd-c258-4b49-8612-44b7a83122c7" (UID: "f162b4dd-c258-4b49-8612-44b7a83122c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:27:38.888723 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:38.888696 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f162b4dd-c258-4b49-8612-44b7a83122c7-maas-api-xks-ca" pod "f162b4dd-c258-4b49-8612-44b7a83122c7" (UID: "f162b4dd-c258-4b49-8612-44b7a83122c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:27:38.888859 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:38.888842 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f162b4dd-c258-4b49-8612-44b7a83122c7-kube-api-access-lb7g2" pod "f162b4dd-c258-4b49-8612-44b7a83122c7" (UID: "f162b4dd-c258-4b49-8612-44b7a83122c7"). InnerVolumeSpecName "kube-api-access-lb7g2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:27:38.988821 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:38.988779 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f162b4dd-c258-4b49-8612-44b7a83122c7-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:27:38.988821 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:38.988814 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lb7g2\" (UniqueName: \"kubernetes.io/projected/f162b4dd-c258-4b49-8612-44b7a83122c7-kube-api-access-lb7g2\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:27:38.988997 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:38.988832 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f162b4dd-c258-4b49-8612-44b7a83122c7-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:27:39.007331 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:39.007297 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-547082/auth-test-model-547082-kserve-7fc64f4477-8cjgx"] Oct 30 20:27:39.631776 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:39.631729 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48526fc3-g7k64" event={"ID":"f162b4dd-c258-4b49-8612-44b7a83122c7","Type":"ContainerDied","Data":"40533cbb3bdf485e9a2bf1d45f376385702ab5d227ed796531c1b4e81860d991"} Oct 30 20:27:39.632377 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:39.632337 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40533cbb3bdf485e9a2bf1d45f376385702ab5d227ed796531c1b4e81860d991" Oct 30 20:27:39.632612 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:39.632587 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48526fc3-g7k64" Oct 30 20:27:43.220273 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:27:43.220231 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:27:45.266209 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:45.266177 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35ecd54c-2jb5d"] Oct 30 20:27:45.266632 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:45.266620 4292 cpu_manager.go:414] "containerMap: removing container" podUID="f18d86a4-efcc-4bb0-91c0-c98e1765acbb" containerName="revoke-keys" Oct 30 20:27:45.266675 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:45.266634 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f18d86a4-efcc-4bb0-91c0-c98e1765acbb" containerName="revoke-keys" Oct 30 20:27:45.266675 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:45.266666 4292 cpu_manager.go:414] "containerMap: removing container" podUID="f162b4dd-c258-4b49-8612-44b7a83122c7" containerName="revoke-keys" Oct 30 20:27:45.266740 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:45.266675 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f162b4dd-c258-4b49-8612-44b7a83122c7" containerName="revoke-keys" Oct 30 20:27:45.266805 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:45.266793 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="f18d86a4-efcc-4bb0-91c0-c98e1765acbb" containerName="revoke-keys" Oct 30 20:27:45.266841 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:45.266808 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="f162b4dd-c258-4b49-8612-44b7a83122c7" containerName="revoke-keys" Oct 30 20:27:45.271221 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:45.271204 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ecd54c-2jb5d" Oct 30 20:27:45.327603 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:45.327570 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35ecd54c-2jb5d"] Oct 30 20:27:45.358635 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:45.358607 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16d5a450-60c4-4075-8f0f-a5d628d7972d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35ecd54c-2jb5d\" (UID: \"16d5a450-60c4-4075-8f0f-a5d628d7972d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ecd54c-2jb5d" Oct 30 20:27:45.358788 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:45.358650 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4bd2\" (UniqueName: \"kubernetes.io/projected/16d5a450-60c4-4075-8f0f-a5d628d7972d-kube-api-access-c4bd2\") pod \"maas-api-revoke-sub-35ecd54c-2jb5d\" (UID: \"16d5a450-60c4-4075-8f0f-a5d628d7972d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ecd54c-2jb5d" Oct 30 20:27:45.358788 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:45.358776 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16d5a450-60c4-4075-8f0f-a5d628d7972d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35ecd54c-2jb5d\" (UID: \"16d5a450-60c4-4075-8f0f-a5d628d7972d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ecd54c-2jb5d" Oct 30 20:27:45.375630 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:45.375605 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:27:45.375780 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:45.375690 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\"" Oct 30 20:27:45.459450 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:45.459404 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16d5a450-60c4-4075-8f0f-a5d628d7972d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35ecd54c-2jb5d\" (UID: \"16d5a450-60c4-4075-8f0f-a5d628d7972d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ecd54c-2jb5d" Oct 30 20:27:45.459629 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:45.459580 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16d5a450-60c4-4075-8f0f-a5d628d7972d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35ecd54c-2jb5d\" (UID: \"16d5a450-60c4-4075-8f0f-a5d628d7972d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ecd54c-2jb5d" Oct 30 20:27:45.459685 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:45.459633 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4bd2\" (UniqueName: \"kubernetes.io/projected/16d5a450-60c4-4075-8f0f-a5d628d7972d-kube-api-access-c4bd2\") pod \"maas-api-revoke-sub-35ecd54c-2jb5d\" (UID: \"16d5a450-60c4-4075-8f0f-a5d628d7972d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ecd54c-2jb5d" Oct 30 20:27:45.460289 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:45.460270 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16d5a450-60c4-4075-8f0f-a5d628d7972d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35ecd54c-2jb5d\" (UID: \"16d5a450-60c4-4075-8f0f-a5d628d7972d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ecd54c-2jb5d" Oct 30 20:27:45.461857 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:45.461837 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16d5a450-60c4-4075-8f0f-a5d628d7972d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35ecd54c-2jb5d\" (UID: \"16d5a450-60c4-4075-8f0f-a5d628d7972d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ecd54c-2jb5d" Oct 30 20:27:45.522567 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:45.522542 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4bd2\" (UniqueName: \"kubernetes.io/projected/16d5a450-60c4-4075-8f0f-a5d628d7972d-kube-api-access-c4bd2\") pod \"maas-api-revoke-sub-35ecd54c-2jb5d\" (UID: \"16d5a450-60c4-4075-8f0f-a5d628d7972d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ecd54c-2jb5d" Oct 30 20:27:45.584928 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:45.584899 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ecd54c-2jb5d" Oct 30 20:27:45.888589 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:45.888514 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dda0de5c-7pm6d"] Oct 30 20:27:45.896768 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:45.896742 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dda0de5c-7pm6d" Oct 30 20:27:45.950905 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:45.950876 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dda0de5c-7pm6d"] Oct 30 20:27:45.965087 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:45.965057 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9t6p\" (UniqueName: \"kubernetes.io/projected/2ee60a2b-7f5f-493a-ad4a-bdd51ae6eefb-kube-api-access-f9t6p\") pod \"maas-api-revoke-sub-dda0de5c-7pm6d\" (UID: \"2ee60a2b-7f5f-493a-ad4a-bdd51ae6eefb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dda0de5c-7pm6d" Oct 30 20:27:45.965272 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:45.965117 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ee60a2b-7f5f-493a-ad4a-bdd51ae6eefb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dda0de5c-7pm6d\" (UID: \"2ee60a2b-7f5f-493a-ad4a-bdd51ae6eefb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dda0de5c-7pm6d" Oct 30 20:27:45.965272 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:45.965226 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ee60a2b-7f5f-493a-ad4a-bdd51ae6eefb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dda0de5c-7pm6d\" (UID: \"2ee60a2b-7f5f-493a-ad4a-bdd51ae6eefb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dda0de5c-7pm6d" Oct 30 20:27:46.004816 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:46.004787 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35ecd54c-2jb5d"] Oct 30 20:27:46.006572 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:27:46.006548 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16d5a450_60c4_4075_8f0f_a5d628d7972d.slice/crio-27063a1c31251efd12f4e01f1aa05cedab037cf6cee1e893763f94dd19ae0d9a WatchSource:0}: Error finding container 27063a1c31251efd12f4e01f1aa05cedab037cf6cee1e893763f94dd19ae0d9a: Status 404 returned error can't find the container with id 27063a1c31251efd12f4e01f1aa05cedab037cf6cee1e893763f94dd19ae0d9a Oct 30 20:27:46.066242 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:46.066203 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9t6p\" (UniqueName: \"kubernetes.io/projected/2ee60a2b-7f5f-493a-ad4a-bdd51ae6eefb-kube-api-access-f9t6p\") pod \"maas-api-revoke-sub-dda0de5c-7pm6d\" (UID: \"2ee60a2b-7f5f-493a-ad4a-bdd51ae6eefb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dda0de5c-7pm6d" Oct 30 20:27:46.066436 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:46.066289 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ee60a2b-7f5f-493a-ad4a-bdd51ae6eefb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dda0de5c-7pm6d\" (UID: \"2ee60a2b-7f5f-493a-ad4a-bdd51ae6eefb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dda0de5c-7pm6d" Oct 30 20:27:46.066436 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:46.066333 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ee60a2b-7f5f-493a-ad4a-bdd51ae6eefb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dda0de5c-7pm6d\" (UID: \"2ee60a2b-7f5f-493a-ad4a-bdd51ae6eefb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dda0de5c-7pm6d" Oct 30 20:27:46.067053 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:46.067032 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ee60a2b-7f5f-493a-ad4a-bdd51ae6eefb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dda0de5c-7pm6d\" (UID: \"2ee60a2b-7f5f-493a-ad4a-bdd51ae6eefb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dda0de5c-7pm6d" Oct 30 20:27:46.068664 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:46.068647 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ee60a2b-7f5f-493a-ad4a-bdd51ae6eefb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dda0de5c-7pm6d\" (UID: \"2ee60a2b-7f5f-493a-ad4a-bdd51ae6eefb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dda0de5c-7pm6d" Oct 30 20:27:46.128187 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:46.128157 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9t6p\" (UniqueName: \"kubernetes.io/projected/2ee60a2b-7f5f-493a-ad4a-bdd51ae6eefb-kube-api-access-f9t6p\") pod \"maas-api-revoke-sub-dda0de5c-7pm6d\" (UID: \"2ee60a2b-7f5f-493a-ad4a-bdd51ae6eefb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dda0de5c-7pm6d" Oct 30 20:27:46.212370 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:46.212316 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dda0de5c-7pm6d" Oct 30 20:27:46.602241 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:27:46.602202 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ee60a2b_7f5f_493a_ad4a_bdd51ae6eefb.slice/crio-d162872511e5ff2f296917d0207af68edd0770cce0831dbf00c685b98b2c3c26 WatchSource:0}: Error finding container d162872511e5ff2f296917d0207af68edd0770cce0831dbf00c685b98b2c3c26: Status 404 returned error can't find the container with id d162872511e5ff2f296917d0207af68edd0770cce0831dbf00c685b98b2c3c26 Oct 30 20:27:46.604387 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:46.604363 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dda0de5c-7pm6d"] Oct 30 20:27:46.672760 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:46.672730 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dda0de5c-7pm6d" event={"ID":"2ee60a2b-7f5f-493a-ad4a-bdd51ae6eefb","Type":"ContainerStarted","Data":"d162872511e5ff2f296917d0207af68edd0770cce0831dbf00c685b98b2c3c26"} Oct 30 20:27:46.674231 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:46.674199 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ecd54c-2jb5d" event={"ID":"16d5a450-60c4-4075-8f0f-a5d628d7972d","Type":"ContainerStarted","Data":"d6647ca481fcc183fe8808378d5fde26761af095cfa501006d4af20417da7927"} Oct 30 20:27:46.674338 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:46.674245 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ecd54c-2jb5d" event={"ID":"16d5a450-60c4-4075-8f0f-a5d628d7972d","Type":"ContainerStarted","Data":"27063a1c31251efd12f4e01f1aa05cedab037cf6cee1e893763f94dd19ae0d9a"} Oct 30 20:27:46.798509 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:46.798459 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ecd54c-2jb5d" podStartSLOduration=1.798445429 podStartE2EDuration="1.798445429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:27:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:27:46.797401556 +0000 UTC m=+3770.315717014" watchObservedRunningTime="2026-10-30 20:27:46.798445429 +0000 UTC m=+3770.316760908" Oct 30 20:27:47.681096 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:47.681012 4292 generic.go:356] "Generic (PLEG): container finished" podID="2ee60a2b-7f5f-493a-ad4a-bdd51ae6eefb" containerID="bec0195d457782ff6a7ec8d79bc348db47ca2fddc6796cacdea78dc34d74bf90" exitCode=0 Oct 30 20:27:47.681540 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:47.681100 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dda0de5c-7pm6d" event={"ID":"2ee60a2b-7f5f-493a-ad4a-bdd51ae6eefb","Type":"ContainerDied","Data":"bec0195d457782ff6a7ec8d79bc348db47ca2fddc6796cacdea78dc34d74bf90"} Oct 30 20:27:47.682595 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:47.682574 4292 generic.go:356] "Generic (PLEG): container finished" podID="16d5a450-60c4-4075-8f0f-a5d628d7972d" containerID="d6647ca481fcc183fe8808378d5fde26761af095cfa501006d4af20417da7927" exitCode=0 Oct 30 20:27:47.682718 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:47.682641 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ecd54c-2jb5d" event={"ID":"16d5a450-60c4-4075-8f0f-a5d628d7972d","Type":"ContainerDied","Data":"d6647ca481fcc183fe8808378d5fde26761af095cfa501006d4af20417da7927"} Oct 30 20:27:48.840867 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:48.840842 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dda0de5c-7pm6d" Oct 30 20:27:48.881953 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:48.881931 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ecd54c-2jb5d" Oct 30 20:27:48.998149 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:48.998065 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ee60a2b-7f5f-493a-ad4a-bdd51ae6eefb-kube-api-access-f9t6p\" (UniqueName: \"kubernetes.io/projected/2ee60a2b-7f5f-493a-ad4a-bdd51ae6eefb-kube-api-access-f9t6p\") pod \"2ee60a2b-7f5f-493a-ad4a-bdd51ae6eefb\" (UID: \"2ee60a2b-7f5f-493a-ad4a-bdd51ae6eefb\") " Oct 30 20:27:48.998149 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:48.998130 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16d5a450-60c4-4075-8f0f-a5d628d7972d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16d5a450-60c4-4075-8f0f-a5d628d7972d-maas-api-xks-ca\") pod \"16d5a450-60c4-4075-8f0f-a5d628d7972d\" (UID: \"16d5a450-60c4-4075-8f0f-a5d628d7972d\") " Oct 30 20:27:48.998407 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:48.998245 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ee60a2b-7f5f-493a-ad4a-bdd51ae6eefb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ee60a2b-7f5f-493a-ad4a-bdd51ae6eefb-maas-api-xks-ca\") pod \"2ee60a2b-7f5f-493a-ad4a-bdd51ae6eefb\" (UID: \"2ee60a2b-7f5f-493a-ad4a-bdd51ae6eefb\") " Oct 30 20:27:48.998407 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:48.998284 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ee60a2b-7f5f-493a-ad4a-bdd51ae6eefb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ee60a2b-7f5f-493a-ad4a-bdd51ae6eefb-maas-api-openshift-ca\") pod \"2ee60a2b-7f5f-493a-ad4a-bdd51ae6eefb\" (UID: \"2ee60a2b-7f5f-493a-ad4a-bdd51ae6eefb\") " Oct 30 20:27:48.998407 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:48.998343 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16d5a450-60c4-4075-8f0f-a5d628d7972d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16d5a450-60c4-4075-8f0f-a5d628d7972d-maas-api-openshift-ca\") pod \"16d5a450-60c4-4075-8f0f-a5d628d7972d\" (UID: \"16d5a450-60c4-4075-8f0f-a5d628d7972d\") " Oct 30 20:27:48.998590 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:48.998444 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16d5a450-60c4-4075-8f0f-a5d628d7972d-kube-api-access-c4bd2\" (UniqueName: \"kubernetes.io/projected/16d5a450-60c4-4075-8f0f-a5d628d7972d-kube-api-access-c4bd2\") pod \"16d5a450-60c4-4075-8f0f-a5d628d7972d\" (UID: \"16d5a450-60c4-4075-8f0f-a5d628d7972d\") " Oct 30 20:27:48.998866 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:48.998834 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16d5a450-60c4-4075-8f0f-a5d628d7972d-maas-api-openshift-ca" pod "16d5a450-60c4-4075-8f0f-a5d628d7972d" (UID: "16d5a450-60c4-4075-8f0f-a5d628d7972d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:27:48.998985 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:48.998954 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16d5a450-60c4-4075-8f0f-a5d628d7972d-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:27:48.998985 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:48.998963 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ee60a2b-7f5f-493a-ad4a-bdd51ae6eefb-maas-api-openshift-ca" pod "2ee60a2b-7f5f-493a-ad4a-bdd51ae6eefb" (UID: "2ee60a2b-7f5f-493a-ad4a-bdd51ae6eefb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:27:49.000551 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:49.000525 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ee60a2b-7f5f-493a-ad4a-bdd51ae6eefb-maas-api-xks-ca" pod "2ee60a2b-7f5f-493a-ad4a-bdd51ae6eefb" (UID: "2ee60a2b-7f5f-493a-ad4a-bdd51ae6eefb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:27:49.000781 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:49.000752 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16d5a450-60c4-4075-8f0f-a5d628d7972d-maas-api-xks-ca" pod "16d5a450-60c4-4075-8f0f-a5d628d7972d" (UID: "16d5a450-60c4-4075-8f0f-a5d628d7972d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:27:49.000781 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:49.000767 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16d5a450-60c4-4075-8f0f-a5d628d7972d-kube-api-access-c4bd2" pod "16d5a450-60c4-4075-8f0f-a5d628d7972d" (UID: "16d5a450-60c4-4075-8f0f-a5d628d7972d"). InnerVolumeSpecName "kube-api-access-c4bd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:27:49.000919 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:49.000903 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ee60a2b-7f5f-493a-ad4a-bdd51ae6eefb-kube-api-access-f9t6p" pod "2ee60a2b-7f5f-493a-ad4a-bdd51ae6eefb" (UID: "2ee60a2b-7f5f-493a-ad4a-bdd51ae6eefb"). InnerVolumeSpecName "kube-api-access-f9t6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:27:49.100176 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:49.100137 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4bd2\" (UniqueName: \"kubernetes.io/projected/16d5a450-60c4-4075-8f0f-a5d628d7972d-kube-api-access-c4bd2\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:27:49.100339 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:49.100179 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9t6p\" (UniqueName: \"kubernetes.io/projected/2ee60a2b-7f5f-493a-ad4a-bdd51ae6eefb-kube-api-access-f9t6p\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:27:49.100339 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:49.100198 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16d5a450-60c4-4075-8f0f-a5d628d7972d-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:27:49.100339 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:49.100213 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ee60a2b-7f5f-493a-ad4a-bdd51ae6eefb-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:27:49.100339 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:49.100227 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ee60a2b-7f5f-493a-ad4a-bdd51ae6eefb-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:27:49.699926 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:49.699899 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dda0de5c-7pm6d" Oct 30 20:27:49.699926 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:49.699905 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dda0de5c-7pm6d" event={"ID":"2ee60a2b-7f5f-493a-ad4a-bdd51ae6eefb","Type":"ContainerDied","Data":"d162872511e5ff2f296917d0207af68edd0770cce0831dbf00c685b98b2c3c26"} Oct 30 20:27:49.700171 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:49.699971 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d162872511e5ff2f296917d0207af68edd0770cce0831dbf00c685b98b2c3c26" Oct 30 20:27:49.701456 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:49.701434 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ecd54c-2jb5d" Oct 30 20:27:49.701615 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:49.701420 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ecd54c-2jb5d" event={"ID":"16d5a450-60c4-4075-8f0f-a5d628d7972d","Type":"ContainerDied","Data":"27063a1c31251efd12f4e01f1aa05cedab037cf6cee1e893763f94dd19ae0d9a"} Oct 30 20:27:49.701615 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:49.701541 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27063a1c31251efd12f4e01f1aa05cedab037cf6cee1e893763f94dd19ae0d9a" Oct 30 20:27:54.027972 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:54.027924 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-547082/auth-test-model-547082-kserve-7fc64f4477-8cjgx" podUID="889dfa94-11f4-425b-8773-2ad89fd14368" containerName="main" containerID="cri-o://6a604410c18ad44007316a7122cd90d6a57c655bd25ff054597291a442b999ec" gracePeriod=45 Oct 30 20:27:54.304483 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:54.304456 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-547082/auth-test-model-547082-kserve-7fc64f4477-8cjgx" Oct 30 20:27:54.462937 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:54.462907 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/889dfa94-11f4-425b-8773-2ad89fd14368-kube-api-access-jpf6x\" (UniqueName: \"kubernetes.io/projected/889dfa94-11f4-425b-8773-2ad89fd14368-kube-api-access-jpf6x\") pod \"889dfa94-11f4-425b-8773-2ad89fd14368\" (UID: \"889dfa94-11f4-425b-8773-2ad89fd14368\") " Oct 30 20:27:54.463107 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:54.462961 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/889dfa94-11f4-425b-8773-2ad89fd14368-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/889dfa94-11f4-425b-8773-2ad89fd14368-model-cache\") pod \"889dfa94-11f4-425b-8773-2ad89fd14368\" (UID: \"889dfa94-11f4-425b-8773-2ad89fd14368\") " Oct 30 20:27:54.463107 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:54.462995 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/889dfa94-11f4-425b-8773-2ad89fd14368-home\" (UniqueName: \"kubernetes.io/empty-dir/889dfa94-11f4-425b-8773-2ad89fd14368-home\") pod \"889dfa94-11f4-425b-8773-2ad89fd14368\" (UID: \"889dfa94-11f4-425b-8773-2ad89fd14368\") " Oct 30 20:27:54.463238 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:54.463143 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/889dfa94-11f4-425b-8773-2ad89fd14368-dshm\" (UniqueName: \"kubernetes.io/empty-dir/889dfa94-11f4-425b-8773-2ad89fd14368-dshm\") pod \"889dfa94-11f4-425b-8773-2ad89fd14368\" (UID: \"889dfa94-11f4-425b-8773-2ad89fd14368\") " Oct 30 20:27:54.463294 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:54.463244 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/889dfa94-11f4-425b-8773-2ad89fd14368-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/889dfa94-11f4-425b-8773-2ad89fd14368-tmp-dir\") pod \"889dfa94-11f4-425b-8773-2ad89fd14368\" (UID: \"889dfa94-11f4-425b-8773-2ad89fd14368\") " Oct 30 20:27:54.463294 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:54.463264 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/889dfa94-11f4-425b-8773-2ad89fd14368-model-cache" pod "889dfa94-11f4-425b-8773-2ad89fd14368" (UID: "889dfa94-11f4-425b-8773-2ad89fd14368"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:27:54.463294 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:54.463271 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/889dfa94-11f4-425b-8773-2ad89fd14368-home" pod "889dfa94-11f4-425b-8773-2ad89fd14368" (UID: "889dfa94-11f4-425b-8773-2ad89fd14368"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:27:54.463488 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:54.463401 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/889dfa94-11f4-425b-8773-2ad89fd14368-tls-certs\" (UniqueName: \"kubernetes.io/secret/889dfa94-11f4-425b-8773-2ad89fd14368-tls-certs\") pod \"889dfa94-11f4-425b-8773-2ad89fd14368\" (UID: \"889dfa94-11f4-425b-8773-2ad89fd14368\") " Oct 30 20:27:54.463585 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:54.463552 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/889dfa94-11f4-425b-8773-2ad89fd14368-tmp-dir" pod "889dfa94-11f4-425b-8773-2ad89fd14368" (UID: "889dfa94-11f4-425b-8773-2ad89fd14368"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:27:54.463881 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:54.463846 4292 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/889dfa94-11f4-425b-8773-2ad89fd14368-home\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:27:54.463881 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:54.463869 4292 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/889dfa94-11f4-425b-8773-2ad89fd14368-tmp-dir\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:27:54.463997 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:54.463882 4292 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/889dfa94-11f4-425b-8773-2ad89fd14368-model-cache\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:27:54.465439 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:54.465417 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/889dfa94-11f4-425b-8773-2ad89fd14368-kube-api-access-jpf6x" pod "889dfa94-11f4-425b-8773-2ad89fd14368" (UID: "889dfa94-11f4-425b-8773-2ad89fd14368"). InnerVolumeSpecName "kube-api-access-jpf6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:27:54.465548 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:54.465415 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/889dfa94-11f4-425b-8773-2ad89fd14368-tls-certs" pod "889dfa94-11f4-425b-8773-2ad89fd14368" (UID: "889dfa94-11f4-425b-8773-2ad89fd14368"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:27:54.465548 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:54.465422 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/889dfa94-11f4-425b-8773-2ad89fd14368-dshm" pod "889dfa94-11f4-425b-8773-2ad89fd14368" (UID: "889dfa94-11f4-425b-8773-2ad89fd14368"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:27:54.564932 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:54.564900 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpf6x\" (UniqueName: \"kubernetes.io/projected/889dfa94-11f4-425b-8773-2ad89fd14368-kube-api-access-jpf6x\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:27:54.564932 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:54.564927 4292 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/889dfa94-11f4-425b-8773-2ad89fd14368-dshm\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:27:54.564932 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:54.564937 4292 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/889dfa94-11f4-425b-8773-2ad89fd14368-tls-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:27:54.743271 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:54.743236 4292 generic.go:356] "Generic (PLEG): container finished" podID="889dfa94-11f4-425b-8773-2ad89fd14368" containerID="6a604410c18ad44007316a7122cd90d6a57c655bd25ff054597291a442b999ec" exitCode=0 Oct 30 20:27:54.743483 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:54.743279 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-547082/auth-test-model-547082-kserve-7fc64f4477-8cjgx" Oct 30 20:27:54.743483 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:54.743320 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-547082/auth-test-model-547082-kserve-7fc64f4477-8cjgx" event={"ID":"889dfa94-11f4-425b-8773-2ad89fd14368","Type":"ContainerDied","Data":"6a604410c18ad44007316a7122cd90d6a57c655bd25ff054597291a442b999ec"} Oct 30 20:27:54.743483 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:54.743384 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-547082/auth-test-model-547082-kserve-7fc64f4477-8cjgx" event={"ID":"889dfa94-11f4-425b-8773-2ad89fd14368","Type":"ContainerDied","Data":"4559c2a0d411703428b936256f00c2831518726ac679591ee8b7972b374a1328"} Oct 30 20:27:54.743483 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:54.743406 4292 scope.go:122] "RemoveContainer" containerID="6a604410c18ad44007316a7122cd90d6a57c655bd25ff054597291a442b999ec" Oct 30 20:27:54.752633 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:54.752616 4292 scope.go:122] "RemoveContainer" containerID="6a604410c18ad44007316a7122cd90d6a57c655bd25ff054597291a442b999ec" Oct 30 20:27:54.752941 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:27:54.752914 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a604410c18ad44007316a7122cd90d6a57c655bd25ff054597291a442b999ec\": container with ID starting with 6a604410c18ad44007316a7122cd90d6a57c655bd25ff054597291a442b999ec not found: ID does not exist" containerID="6a604410c18ad44007316a7122cd90d6a57c655bd25ff054597291a442b999ec" Oct 30 20:27:54.752997 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:54.752947 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a604410c18ad44007316a7122cd90d6a57c655bd25ff054597291a442b999ec"} err="failed to get container status \"6a604410c18ad44007316a7122cd90d6a57c655bd25ff054597291a442b999ec\": rpc error: code = NotFound desc = could not find container \"6a604410c18ad44007316a7122cd90d6a57c655bd25ff054597291a442b999ec\": container with ID starting with 6a604410c18ad44007316a7122cd90d6a57c655bd25ff054597291a442b999ec not found: ID does not exist" Oct 30 20:27:54.912104 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:54.912020 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-547082/auth-test-model-547082-kserve-7fc64f4477-8cjgx"] Oct 30 20:27:54.916467 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:54.916429 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-547082/auth-test-model-547082-kserve-7fc64f4477-8cjgx"] Oct 30 20:27:56.586502 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:27:56.586468 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="889dfa94-11f4-425b-8773-2ad89fd14368" path="/var/lib/kubelet/pods/889dfa94-11f4-425b-8773-2ad89fd14368/volumes" Oct 30 20:28:09.811837 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:09.811802 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds"] Oct 30 20:28:09.812483 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:09.812461 4292 cpu_manager.go:414] "containerMap: removing container" podUID="2ee60a2b-7f5f-493a-ad4a-bdd51ae6eefb" containerName="revoke-keys" Oct 30 20:28:09.812483 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:09.812482 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ee60a2b-7f5f-493a-ad4a-bdd51ae6eefb" containerName="revoke-keys" Oct 30 20:28:09.812661 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:09.812521 4292 cpu_manager.go:414] "containerMap: removing container" podUID="16d5a450-60c4-4075-8f0f-a5d628d7972d" containerName="revoke-keys" Oct 30 20:28:09.812661 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:09.812530 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16d5a450-60c4-4075-8f0f-a5d628d7972d" containerName="revoke-keys" Oct 30 20:28:09.812661 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:09.812563 4292 cpu_manager.go:414] "containerMap: removing container" podUID="889dfa94-11f4-425b-8773-2ad89fd14368" containerName="main" Oct 30 20:28:09.812661 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:09.812570 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="889dfa94-11f4-425b-8773-2ad89fd14368" containerName="main" Oct 30 20:28:09.812859 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:09.812676 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="889dfa94-11f4-425b-8773-2ad89fd14368" containerName="main" Oct 30 20:28:09.812859 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:09.812692 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ee60a2b-7f5f-493a-ad4a-bdd51ae6eefb" containerName="revoke-keys" Oct 30 20:28:09.812859 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:09.812706 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="16d5a450-60c4-4075-8f0f-a5d628d7972d" containerName="revoke-keys" Oct 30 20:28:09.816049 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:09.816028 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds" Oct 30 20:28:09.885373 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:09.883929 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-0e72bd-openshift-default-dockercfg-tr5gd\"" Oct 30 20:28:09.943087 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:09.943051 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03c3a9f1-5156-41a2-a643-46343f91b78e-workload-certs\") pod \"e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds\" (UID: \"03c3a9f1-5156-41a2-a643-46343f91b78e\") " pod="openshift-ingress/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds" Oct 30 20:28:09.943377 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:09.943332 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03c3a9f1-5156-41a2-a643-46343f91b78e-istio-podinfo\") pod \"e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds\" (UID: \"03c3a9f1-5156-41a2-a643-46343f91b78e\") " pod="openshift-ingress/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds" Oct 30 20:28:09.943614 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:09.943587 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03c3a9f1-5156-41a2-a643-46343f91b78e-credential-socket\") pod \"e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds\" (UID: \"03c3a9f1-5156-41a2-a643-46343f91b78e\") " pod="openshift-ingress/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds" Oct 30 20:28:09.943759 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:09.943743 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03c3a9f1-5156-41a2-a643-46343f91b78e-istio-token\") pod \"e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds\" (UID: \"03c3a9f1-5156-41a2-a643-46343f91b78e\") " pod="openshift-ingress/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds" Oct 30 20:28:09.943918 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:09.943904 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03c3a9f1-5156-41a2-a643-46343f91b78e-istio-envoy\") pod \"e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds\" (UID: \"03c3a9f1-5156-41a2-a643-46343f91b78e\") " pod="openshift-ingress/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds" Oct 30 20:28:09.944079 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:09.944050 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03c3a9f1-5156-41a2-a643-46343f91b78e-istiod-ca-cert\") pod \"e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds\" (UID: \"03c3a9f1-5156-41a2-a643-46343f91b78e\") " pod="openshift-ingress/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds" Oct 30 20:28:09.944245 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:09.944224 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03c3a9f1-5156-41a2-a643-46343f91b78e-istio-data\") pod \"e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds\" (UID: \"03c3a9f1-5156-41a2-a643-46343f91b78e\") " pod="openshift-ingress/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds" Oct 30 20:28:09.944411 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:09.944387 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03c3a9f1-5156-41a2-a643-46343f91b78e-workload-socket\") pod \"e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds\" (UID: \"03c3a9f1-5156-41a2-a643-46343f91b78e\") " pod="openshift-ingress/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds" Oct 30 20:28:09.944491 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:09.944422 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2pd4\" (UniqueName: \"kubernetes.io/projected/03c3a9f1-5156-41a2-a643-46343f91b78e-kube-api-access-j2pd4\") pod \"e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds\" (UID: \"03c3a9f1-5156-41a2-a643-46343f91b78e\") " pod="openshift-ingress/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds" Oct 30 20:28:09.945594 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:09.945571 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds"] Oct 30 20:28:10.046063 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:10.046020 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03c3a9f1-5156-41a2-a643-46343f91b78e-credential-socket\") pod \"e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds\" (UID: \"03c3a9f1-5156-41a2-a643-46343f91b78e\") " pod="openshift-ingress/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds" Oct 30 20:28:10.046286 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:10.046270 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03c3a9f1-5156-41a2-a643-46343f91b78e-istio-token\") pod \"e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds\" (UID: \"03c3a9f1-5156-41a2-a643-46343f91b78e\") " pod="openshift-ingress/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds" Oct 30 20:28:10.046617 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:10.046595 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03c3a9f1-5156-41a2-a643-46343f91b78e-istio-envoy\") pod \"e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds\" (UID: \"03c3a9f1-5156-41a2-a643-46343f91b78e\") " pod="openshift-ingress/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds" Oct 30 20:28:10.047543 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:10.047519 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03c3a9f1-5156-41a2-a643-46343f91b78e-istiod-ca-cert\") pod \"e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds\" (UID: \"03c3a9f1-5156-41a2-a643-46343f91b78e\") " pod="openshift-ingress/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds" Oct 30 20:28:10.047706 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:10.047689 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03c3a9f1-5156-41a2-a643-46343f91b78e-istio-data\") pod \"e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds\" (UID: \"03c3a9f1-5156-41a2-a643-46343f91b78e\") " pod="openshift-ingress/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds" Oct 30 20:28:10.047847 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:10.047832 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03c3a9f1-5156-41a2-a643-46343f91b78e-workload-socket\") pod \"e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds\" (UID: \"03c3a9f1-5156-41a2-a643-46343f91b78e\") " pod="openshift-ingress/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds" Oct 30 20:28:10.047953 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:10.047936 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2pd4\" (UniqueName: \"kubernetes.io/projected/03c3a9f1-5156-41a2-a643-46343f91b78e-kube-api-access-j2pd4\") pod \"e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds\" (UID: \"03c3a9f1-5156-41a2-a643-46343f91b78e\") " pod="openshift-ingress/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds" Oct 30 20:28:10.048082 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:10.048067 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03c3a9f1-5156-41a2-a643-46343f91b78e-workload-certs\") pod \"e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds\" (UID: \"03c3a9f1-5156-41a2-a643-46343f91b78e\") " pod="openshift-ingress/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds" Oct 30 20:28:10.048191 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:10.048179 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03c3a9f1-5156-41a2-a643-46343f91b78e-istio-podinfo\") pod \"e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds\" (UID: \"03c3a9f1-5156-41a2-a643-46343f91b78e\") " pod="openshift-ingress/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds" Oct 30 20:28:10.048677 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:10.048650 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03c3a9f1-5156-41a2-a643-46343f91b78e-istiod-ca-cert\") pod \"e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds\" (UID: \"03c3a9f1-5156-41a2-a643-46343f91b78e\") " pod="openshift-ingress/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds" Oct 30 20:28:10.048766 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:10.046818 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03c3a9f1-5156-41a2-a643-46343f91b78e-credential-socket\") pod \"e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds\" (UID: \"03c3a9f1-5156-41a2-a643-46343f91b78e\") " pod="openshift-ingress/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds" Oct 30 20:28:10.048962 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:10.048936 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03c3a9f1-5156-41a2-a643-46343f91b78e-workload-socket\") pod \"e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds\" (UID: \"03c3a9f1-5156-41a2-a643-46343f91b78e\") " pod="openshift-ingress/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds" Oct 30 20:28:10.049222 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:10.049199 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03c3a9f1-5156-41a2-a643-46343f91b78e-istio-data\") pod \"e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds\" (UID: \"03c3a9f1-5156-41a2-a643-46343f91b78e\") " pod="openshift-ingress/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds" Oct 30 20:28:10.055371 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:10.049605 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03c3a9f1-5156-41a2-a643-46343f91b78e-workload-certs\") pod \"e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds\" (UID: \"03c3a9f1-5156-41a2-a643-46343f91b78e\") " pod="openshift-ingress/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds" Oct 30 20:28:10.082266 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:10.082204 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03c3a9f1-5156-41a2-a643-46343f91b78e-istio-podinfo\") pod \"e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds\" (UID: \"03c3a9f1-5156-41a2-a643-46343f91b78e\") " pod="openshift-ingress/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds" Oct 30 20:28:10.085922 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:10.085808 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03c3a9f1-5156-41a2-a643-46343f91b78e-istio-envoy\") pod \"e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds\" (UID: \"03c3a9f1-5156-41a2-a643-46343f91b78e\") " pod="openshift-ingress/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds" Oct 30 20:28:10.113211 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:10.113183 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03c3a9f1-5156-41a2-a643-46343f91b78e-istio-token\") pod \"e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds\" (UID: \"03c3a9f1-5156-41a2-a643-46343f91b78e\") " pod="openshift-ingress/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds" Oct 30 20:28:10.114838 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:10.114817 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2pd4\" (UniqueName: \"kubernetes.io/projected/03c3a9f1-5156-41a2-a643-46343f91b78e-kube-api-access-j2pd4\") pod \"e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds\" (UID: \"03c3a9f1-5156-41a2-a643-46343f91b78e\") " pod="openshift-ingress/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds" Oct 30 20:28:10.129765 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:10.129722 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds" Oct 30 20:28:10.603196 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:10.603167 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds"] Oct 30 20:28:10.608660 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:28:10.608629 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03c3a9f1_5156_41a2_a643_46343f91b78e.slice/crio-718bd3be22fcd0be3bcf822a7fd28fe3b98a38615c16da8b75d3db8b72bf373b WatchSource:0}: Error finding container 718bd3be22fcd0be3bcf822a7fd28fe3b98a38615c16da8b75d3db8b72bf373b: Status 404 returned error can't find the container with id 718bd3be22fcd0be3bcf822a7fd28fe3b98a38615c16da8b75d3db8b72bf373b Oct 30 20:28:10.610792 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:10.610761 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:28:10.610871 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:10.610827 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:28:10.610871 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:10.610858 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:28:10.891376 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:10.891252 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds" event={"ID":"03c3a9f1-5156-41a2-a643-46343f91b78e","Type":"ContainerStarted","Data":"f61b08f353f0f4e288f4c040cd0e9ab5164fe2efc409f8a2bf1c0d2aaecfac32"} Oct 30 20:28:10.891376 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:10.891330 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds" event={"ID":"03c3a9f1-5156-41a2-a643-46343f91b78e","Type":"ContainerStarted","Data":"718bd3be22fcd0be3bcf822a7fd28fe3b98a38615c16da8b75d3db8b72bf373b"} Oct 30 20:28:11.029806 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:11.029745 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds" podStartSLOduration=2.029728926 podStartE2EDuration="2.029728926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:28:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:28:11.026835798 +0000 UTC m=+3794.545151276" watchObservedRunningTime="2026-10-30 20:28:11.029728926 +0000 UTC m=+3794.548044382" Oct 30 20:28:11.130154 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:11.130121 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds" Oct 30 20:28:11.131466 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:11.131440 4292 patch_prober.go:29] interesting pod/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Oct 30 20:28:11.131632 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:11.131483 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds" podUID="03c3a9f1-5156-41a2-a643-46343f91b78e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Oct 30 20:28:12.136782 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:12.136748 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds" Oct 30 20:28:12.905416 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:12.905315 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds" Oct 30 20:28:12.907150 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:12.907116 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds" Oct 30 20:28:13.227027 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:28:13.226937 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:28:14.326539 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:14.326491 4292 patch_prober.go:29] interesting pod/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:28:14.326949 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:14.326569 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl" podUID="7758bd21-5416-4ab0-8c70-59970dc01209" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:28:15.922762 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:15.922659 4292 generic.go:356] "Generic (PLEG): container finished" podID="7758bd21-5416-4ab0-8c70-59970dc01209" containerID="22d0158a2b2bf2aa55272736c8fc7d1c78739f5d782aa32f1def9a5c0a625dbf" exitCode=0 Oct 30 20:28:15.922762 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:15.922731 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl" event={"ID":"7758bd21-5416-4ab0-8c70-59970dc01209","Type":"ContainerDied","Data":"22d0158a2b2bf2aa55272736c8fc7d1c78739f5d782aa32f1def9a5c0a625dbf"} Oct 30 20:28:15.923239 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:15.923136 4292 scope.go:122] "RemoveContainer" containerID="22d0158a2b2bf2aa55272736c8fc7d1c78739f5d782aa32f1def9a5c0a625dbf" Oct 30 20:28:15.924764 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:15.924733 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:28:15.925114 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:15.924808 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:28:15.925114 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:15.924848 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:28:16.930202 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:16.930166 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl" event={"ID":"7758bd21-5416-4ab0-8c70-59970dc01209","Type":"ContainerStarted","Data":"3d02355f58c54ce2360837abf8ca61b881af6f3d03c70941b2712da765dc8999"} Oct 30 20:28:17.690375 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:17.676999 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl" Oct 30 20:28:17.690375 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:17.677117 4292 patch_prober.go:29] interesting pod/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 30 20:28:17.690375 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:17.677159 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl" podUID="7758bd21-5416-4ab0-8c70-59970dc01209" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 30 20:28:18.676667 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:18.676630 4292 patch_prober.go:29] interesting pod/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 30 20:28:18.677081 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:18.676691 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl" podUID="7758bd21-5416-4ab0-8c70-59970dc01209" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 30 20:28:19.677324 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:19.677281 4292 patch_prober.go:29] interesting pod/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 30 20:28:19.677881 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:19.677387 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl" podUID="7758bd21-5416-4ab0-8c70-59970dc01209" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 30 20:28:19.686341 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:19.686309 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-487c5f/auth-test-model-487c5f-kserve-55dfcd6c8f-26x9l"] Oct 30 20:28:20.676801 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:20.676759 4292 patch_prober.go:29] interesting pod/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 30 20:28:20.677026 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:20.676813 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl" podUID="7758bd21-5416-4ab0-8c70-59970dc01209" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 30 20:28:21.677637 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:21.676839 4292 patch_prober.go:29] interesting pod/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 30 20:28:21.677637 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:21.676906 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl" podUID="7758bd21-5416-4ab0-8c70-59970dc01209" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 30 20:28:22.681198 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:22.681166 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl" Oct 30 20:28:22.681651 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:22.681520 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl" Oct 30 20:28:22.682136 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:22.682116 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl" Oct 30 20:28:34.710207 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:34.710150 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-487c5f/auth-test-model-487c5f-kserve-55dfcd6c8f-26x9l" podUID="9295a589-1cb3-4d27-8a26-eb8da390b23a" containerName="main" containerID="cri-o://0d155e8b2332e56b2179ff77e6d935dc28879204234cc9a0be812ad47b71ca1c" gracePeriod=45 Oct 30 20:28:34.968666 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:34.968609 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-487c5f/auth-test-model-487c5f-kserve-55dfcd6c8f-26x9l" Oct 30 20:28:35.057475 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:35.057443 4292 generic.go:356] "Generic (PLEG): container finished" podID="9295a589-1cb3-4d27-8a26-eb8da390b23a" containerID="0d155e8b2332e56b2179ff77e6d935dc28879204234cc9a0be812ad47b71ca1c" exitCode=0 Oct 30 20:28:35.057634 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:35.057485 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-487c5f/auth-test-model-487c5f-kserve-55dfcd6c8f-26x9l" Oct 30 20:28:35.057634 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:35.057525 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-487c5f/auth-test-model-487c5f-kserve-55dfcd6c8f-26x9l" event={"ID":"9295a589-1cb3-4d27-8a26-eb8da390b23a","Type":"ContainerDied","Data":"0d155e8b2332e56b2179ff77e6d935dc28879204234cc9a0be812ad47b71ca1c"} Oct 30 20:28:35.057634 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:35.057565 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-487c5f/auth-test-model-487c5f-kserve-55dfcd6c8f-26x9l" event={"ID":"9295a589-1cb3-4d27-8a26-eb8da390b23a","Type":"ContainerDied","Data":"68f06c2c52548c496ef6173e9fae44c4140dae316b64c332c93ca3a07074c32b"} Oct 30 20:28:35.057634 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:35.057584 4292 scope.go:122] "RemoveContainer" containerID="0d155e8b2332e56b2179ff77e6d935dc28879204234cc9a0be812ad47b71ca1c" Oct 30 20:28:35.066133 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:35.066104 4292 scope.go:122] "RemoveContainer" containerID="0d155e8b2332e56b2179ff77e6d935dc28879204234cc9a0be812ad47b71ca1c" Oct 30 20:28:35.066398 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:28:35.066376 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d155e8b2332e56b2179ff77e6d935dc28879204234cc9a0be812ad47b71ca1c\": container with ID starting with 0d155e8b2332e56b2179ff77e6d935dc28879204234cc9a0be812ad47b71ca1c not found: ID does not exist" containerID="0d155e8b2332e56b2179ff77e6d935dc28879204234cc9a0be812ad47b71ca1c" Oct 30 20:28:35.066501 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:35.066402 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d155e8b2332e56b2179ff77e6d935dc28879204234cc9a0be812ad47b71ca1c"} err="failed to get container status \"0d155e8b2332e56b2179ff77e6d935dc28879204234cc9a0be812ad47b71ca1c\": rpc error: code = NotFound desc = could not find container \"0d155e8b2332e56b2179ff77e6d935dc28879204234cc9a0be812ad47b71ca1c\": container with ID starting with 0d155e8b2332e56b2179ff77e6d935dc28879204234cc9a0be812ad47b71ca1c not found: ID does not exist" Oct 30 20:28:35.106829 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:35.106802 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9295a589-1cb3-4d27-8a26-eb8da390b23a-home\" (UniqueName: \"kubernetes.io/empty-dir/9295a589-1cb3-4d27-8a26-eb8da390b23a-home\") pod \"9295a589-1cb3-4d27-8a26-eb8da390b23a\" (UID: \"9295a589-1cb3-4d27-8a26-eb8da390b23a\") " Oct 30 20:28:35.106969 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:35.106876 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9295a589-1cb3-4d27-8a26-eb8da390b23a-kube-api-access-84crc\" (UniqueName: \"kubernetes.io/projected/9295a589-1cb3-4d27-8a26-eb8da390b23a-kube-api-access-84crc\") pod \"9295a589-1cb3-4d27-8a26-eb8da390b23a\" (UID: \"9295a589-1cb3-4d27-8a26-eb8da390b23a\") " Oct 30 20:28:35.106969 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:35.106907 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9295a589-1cb3-4d27-8a26-eb8da390b23a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9295a589-1cb3-4d27-8a26-eb8da390b23a-dshm\") pod \"9295a589-1cb3-4d27-8a26-eb8da390b23a\" (UID: \"9295a589-1cb3-4d27-8a26-eb8da390b23a\") " Oct 30 20:28:35.107107 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:35.107050 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9295a589-1cb3-4d27-8a26-eb8da390b23a-tls-certs\" (UniqueName: \"kubernetes.io/secret/9295a589-1cb3-4d27-8a26-eb8da390b23a-tls-certs\") pod \"9295a589-1cb3-4d27-8a26-eb8da390b23a\" (UID: \"9295a589-1cb3-4d27-8a26-eb8da390b23a\") " Oct 30 20:28:35.107107 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:35.107079 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9295a589-1cb3-4d27-8a26-eb8da390b23a-home" pod "9295a589-1cb3-4d27-8a26-eb8da390b23a" (UID: "9295a589-1cb3-4d27-8a26-eb8da390b23a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:28:35.107237 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:35.107157 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9295a589-1cb3-4d27-8a26-eb8da390b23a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9295a589-1cb3-4d27-8a26-eb8da390b23a-tmp-dir\") pod \"9295a589-1cb3-4d27-8a26-eb8da390b23a\" (UID: \"9295a589-1cb3-4d27-8a26-eb8da390b23a\") " Oct 30 20:28:35.107306 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:35.107271 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9295a589-1cb3-4d27-8a26-eb8da390b23a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9295a589-1cb3-4d27-8a26-eb8da390b23a-model-cache\") pod \"9295a589-1cb3-4d27-8a26-eb8da390b23a\" (UID: \"9295a589-1cb3-4d27-8a26-eb8da390b23a\") " Oct 30 20:28:35.107621 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:35.107595 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9295a589-1cb3-4d27-8a26-eb8da390b23a-tmp-dir" pod "9295a589-1cb3-4d27-8a26-eb8da390b23a" (UID: "9295a589-1cb3-4d27-8a26-eb8da390b23a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:28:35.107738 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:35.107707 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9295a589-1cb3-4d27-8a26-eb8da390b23a-model-cache" pod "9295a589-1cb3-4d27-8a26-eb8da390b23a" (UID: "9295a589-1cb3-4d27-8a26-eb8da390b23a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:28:35.107814 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:35.107775 4292 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9295a589-1cb3-4d27-8a26-eb8da390b23a-home\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:28:35.107814 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:35.107794 4292 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9295a589-1cb3-4d27-8a26-eb8da390b23a-tmp-dir\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:28:35.109301 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:35.109274 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9295a589-1cb3-4d27-8a26-eb8da390b23a-dshm" pod "9295a589-1cb3-4d27-8a26-eb8da390b23a" (UID: "9295a589-1cb3-4d27-8a26-eb8da390b23a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:28:35.109462 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:35.109306 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9295a589-1cb3-4d27-8a26-eb8da390b23a-tls-certs" pod "9295a589-1cb3-4d27-8a26-eb8da390b23a" (UID: "9295a589-1cb3-4d27-8a26-eb8da390b23a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:28:35.109462 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:35.109387 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9295a589-1cb3-4d27-8a26-eb8da390b23a-kube-api-access-84crc" pod "9295a589-1cb3-4d27-8a26-eb8da390b23a" (UID: "9295a589-1cb3-4d27-8a26-eb8da390b23a"). InnerVolumeSpecName "kube-api-access-84crc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:28:35.208325 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:35.208297 4292 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9295a589-1cb3-4d27-8a26-eb8da390b23a-model-cache\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:28:35.208325 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:35.208326 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84crc\" (UniqueName: \"kubernetes.io/projected/9295a589-1cb3-4d27-8a26-eb8da390b23a-kube-api-access-84crc\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:28:35.208520 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:35.208337 4292 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9295a589-1cb3-4d27-8a26-eb8da390b23a-dshm\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:28:35.208520 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:35.208347 4292 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9295a589-1cb3-4d27-8a26-eb8da390b23a-tls-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:28:35.481388 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:35.481336 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-487c5f/auth-test-model-487c5f-kserve-55dfcd6c8f-26x9l"] Oct 30 20:28:35.485707 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:35.485678 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-487c5f/auth-test-model-487c5f-kserve-55dfcd6c8f-26x9l"] Oct 30 20:28:36.582213 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:36.582179 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9295a589-1cb3-4d27-8a26-eb8da390b23a" path="/var/lib/kubelet/pods/9295a589-1cb3-4d27-8a26-eb8da390b23a/volumes" Oct 30 20:28:41.413063 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:41.412981 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-cafbb2/auth-test-model-cafbb2-kserve-96fc794d9-g6gkh"] Oct 30 20:28:43.219572 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:28:43.219533 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:28:56.432761 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:56.432697 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-cafbb2/auth-test-model-cafbb2-kserve-96fc794d9-g6gkh" podUID="71a33844-736e-4eeb-b9b9-989bd68a6db3" containerName="main" containerID="cri-o://7f24272ab4f605d494abd486721fcc76599bdf6b64150132134460555d5fc087" gracePeriod=45 Oct 30 20:28:56.517605 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:56.517569 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-cafbb2/auth-test-model-cafbb2-kserve-96fc794d9-g6gkh" podUID="71a33844-736e-4eeb-b9b9-989bd68a6db3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.159:8000/ready\": dial tcp 10.133.0.159:8000: connect: connection refused" Oct 30 20:28:56.694017 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:56.693957 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-cafbb2/auth-test-model-cafbb2-kserve-96fc794d9-g6gkh" Oct 30 20:28:56.748429 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:56.748397 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71a33844-736e-4eeb-b9b9-989bd68a6db3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71a33844-736e-4eeb-b9b9-989bd68a6db3-model-cache\") pod \"71a33844-736e-4eeb-b9b9-989bd68a6db3\" (UID: \"71a33844-736e-4eeb-b9b9-989bd68a6db3\") " Oct 30 20:28:56.748599 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:56.748484 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71a33844-736e-4eeb-b9b9-989bd68a6db3-tls-certs\" (UniqueName: \"kubernetes.io/secret/71a33844-736e-4eeb-b9b9-989bd68a6db3-tls-certs\") pod \"71a33844-736e-4eeb-b9b9-989bd68a6db3\" (UID: \"71a33844-736e-4eeb-b9b9-989bd68a6db3\") " Oct 30 20:28:56.748599 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:56.748513 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71a33844-736e-4eeb-b9b9-989bd68a6db3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71a33844-736e-4eeb-b9b9-989bd68a6db3-tmp-dir\") pod \"71a33844-736e-4eeb-b9b9-989bd68a6db3\" (UID: \"71a33844-736e-4eeb-b9b9-989bd68a6db3\") " Oct 30 20:28:56.748599 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:56.748538 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71a33844-736e-4eeb-b9b9-989bd68a6db3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/71a33844-736e-4eeb-b9b9-989bd68a6db3-dshm\") pod \"71a33844-736e-4eeb-b9b9-989bd68a6db3\" (UID: \"71a33844-736e-4eeb-b9b9-989bd68a6db3\") " Oct 30 20:28:56.748784 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:56.748629 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71a33844-736e-4eeb-b9b9-989bd68a6db3-home\" (UniqueName: \"kubernetes.io/empty-dir/71a33844-736e-4eeb-b9b9-989bd68a6db3-home\") pod \"71a33844-736e-4eeb-b9b9-989bd68a6db3\" (UID: \"71a33844-736e-4eeb-b9b9-989bd68a6db3\") " Oct 30 20:28:56.748784 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:56.748647 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71a33844-736e-4eeb-b9b9-989bd68a6db3-kube-api-access-6x7fs\" (UniqueName: \"kubernetes.io/projected/71a33844-736e-4eeb-b9b9-989bd68a6db3-kube-api-access-6x7fs\") pod \"71a33844-736e-4eeb-b9b9-989bd68a6db3\" (UID: \"71a33844-736e-4eeb-b9b9-989bd68a6db3\") " Oct 30 20:28:56.748784 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:56.748669 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a33844-736e-4eeb-b9b9-989bd68a6db3-model-cache" pod "71a33844-736e-4eeb-b9b9-989bd68a6db3" (UID: "71a33844-736e-4eeb-b9b9-989bd68a6db3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:28:56.748952 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:56.748793 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a33844-736e-4eeb-b9b9-989bd68a6db3-tmp-dir" pod "71a33844-736e-4eeb-b9b9-989bd68a6db3" (UID: "71a33844-736e-4eeb-b9b9-989bd68a6db3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:28:56.748952 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:56.748833 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a33844-736e-4eeb-b9b9-989bd68a6db3-home" pod "71a33844-736e-4eeb-b9b9-989bd68a6db3" (UID: "71a33844-736e-4eeb-b9b9-989bd68a6db3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:28:56.749096 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:56.749074 4292 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71a33844-736e-4eeb-b9b9-989bd68a6db3-model-cache\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:28:56.749190 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:56.749102 4292 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71a33844-736e-4eeb-b9b9-989bd68a6db3-tmp-dir\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:28:56.749190 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:56.749116 4292 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71a33844-736e-4eeb-b9b9-989bd68a6db3-home\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:28:56.750861 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:56.750833 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a33844-736e-4eeb-b9b9-989bd68a6db3-dshm" pod "71a33844-736e-4eeb-b9b9-989bd68a6db3" (UID: "71a33844-736e-4eeb-b9b9-989bd68a6db3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:28:56.750982 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:56.750839 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71a33844-736e-4eeb-b9b9-989bd68a6db3-tls-certs" pod "71a33844-736e-4eeb-b9b9-989bd68a6db3" (UID: "71a33844-736e-4eeb-b9b9-989bd68a6db3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:28:56.750982 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:56.750901 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71a33844-736e-4eeb-b9b9-989bd68a6db3-kube-api-access-6x7fs" pod "71a33844-736e-4eeb-b9b9-989bd68a6db3" (UID: "71a33844-736e-4eeb-b9b9-989bd68a6db3"). InnerVolumeSpecName "kube-api-access-6x7fs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:28:56.853406 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:56.853371 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6x7fs\" (UniqueName: \"kubernetes.io/projected/71a33844-736e-4eeb-b9b9-989bd68a6db3-kube-api-access-6x7fs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:28:56.853406 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:56.853412 4292 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71a33844-736e-4eeb-b9b9-989bd68a6db3-tls-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:28:56.853680 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:56.853428 4292 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71a33844-736e-4eeb-b9b9-989bd68a6db3-dshm\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:28:57.185493 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:57.185459 4292 generic.go:356] "Generic (PLEG): container finished" podID="71a33844-736e-4eeb-b9b9-989bd68a6db3" containerID="7f24272ab4f605d494abd486721fcc76599bdf6b64150132134460555d5fc087" exitCode=0 Oct 30 20:28:57.185694 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:57.185501 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-cafbb2/auth-test-model-cafbb2-kserve-96fc794d9-g6gkh" Oct 30 20:28:57.185694 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:57.185550 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-cafbb2/auth-test-model-cafbb2-kserve-96fc794d9-g6gkh" event={"ID":"71a33844-736e-4eeb-b9b9-989bd68a6db3","Type":"ContainerDied","Data":"7f24272ab4f605d494abd486721fcc76599bdf6b64150132134460555d5fc087"} Oct 30 20:28:57.185694 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:57.185590 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-cafbb2/auth-test-model-cafbb2-kserve-96fc794d9-g6gkh" event={"ID":"71a33844-736e-4eeb-b9b9-989bd68a6db3","Type":"ContainerDied","Data":"d146055405d6127e16a36f66c39dc0c34cddff55b4eb307ab6ff58019e838a4f"} Oct 30 20:28:57.185694 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:57.185610 4292 scope.go:122] "RemoveContainer" containerID="7f24272ab4f605d494abd486721fcc76599bdf6b64150132134460555d5fc087" Oct 30 20:28:57.194779 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:57.194761 4292 scope.go:122] "RemoveContainer" containerID="7f24272ab4f605d494abd486721fcc76599bdf6b64150132134460555d5fc087" Oct 30 20:28:57.195032 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:28:57.195014 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f24272ab4f605d494abd486721fcc76599bdf6b64150132134460555d5fc087\": container with ID starting with 7f24272ab4f605d494abd486721fcc76599bdf6b64150132134460555d5fc087 not found: ID does not exist" containerID="7f24272ab4f605d494abd486721fcc76599bdf6b64150132134460555d5fc087" Oct 30 20:28:57.195097 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:57.195040 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f24272ab4f605d494abd486721fcc76599bdf6b64150132134460555d5fc087"} err="failed to get container status \"7f24272ab4f605d494abd486721fcc76599bdf6b64150132134460555d5fc087\": rpc error: code = NotFound desc = could not find container \"7f24272ab4f605d494abd486721fcc76599bdf6b64150132134460555d5fc087\": container with ID starting with 7f24272ab4f605d494abd486721fcc76599bdf6b64150132134460555d5fc087 not found: ID does not exist" Oct 30 20:28:57.324211 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:57.324178 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccbc8582-ljr7s"] Oct 30 20:28:57.324637 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:57.324624 4292 cpu_manager.go:414] "containerMap: removing container" podUID="71a33844-736e-4eeb-b9b9-989bd68a6db3" containerName="main" Oct 30 20:28:57.324696 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:57.324638 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71a33844-736e-4eeb-b9b9-989bd68a6db3" containerName="main" Oct 30 20:28:57.324696 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:57.324653 4292 cpu_manager.go:414] "containerMap: removing container" podUID="9295a589-1cb3-4d27-8a26-eb8da390b23a" containerName="main" Oct 30 20:28:57.324696 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:57.324658 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9295a589-1cb3-4d27-8a26-eb8da390b23a" containerName="main" Oct 30 20:28:57.324790 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:57.324729 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="71a33844-736e-4eeb-b9b9-989bd68a6db3" containerName="main" Oct 30 20:28:57.324790 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:57.324740 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="9295a589-1cb3-4d27-8a26-eb8da390b23a" containerName="main" Oct 30 20:28:57.329414 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:57.329393 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccbc8582-ljr7s" Oct 30 20:28:57.357473 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:57.357439 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-cafbb2/auth-test-model-cafbb2-kserve-96fc794d9-g6gkh"] Oct 30 20:28:57.359993 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:57.359961 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxpwc\" (UniqueName: \"kubernetes.io/projected/65afac6e-94e2-4531-aba0-0355ec17aeb7-kube-api-access-fxpwc\") pod \"maas-api-revoke-sub-ccbc8582-ljr7s\" (UID: \"65afac6e-94e2-4531-aba0-0355ec17aeb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccbc8582-ljr7s" Oct 30 20:28:57.360143 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:57.360022 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65afac6e-94e2-4531-aba0-0355ec17aeb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ccbc8582-ljr7s\" (UID: \"65afac6e-94e2-4531-aba0-0355ec17aeb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccbc8582-ljr7s" Oct 30 20:28:57.360196 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:57.360166 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65afac6e-94e2-4531-aba0-0355ec17aeb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ccbc8582-ljr7s\" (UID: \"65afac6e-94e2-4531-aba0-0355ec17aeb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccbc8582-ljr7s" Oct 30 20:28:57.361388 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:57.361346 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-cafbb2/auth-test-model-cafbb2-kserve-96fc794d9-g6gkh"] Oct 30 20:28:57.383194 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:57.383162 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:28:57.383194 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:57.383189 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\"" Oct 30 20:28:57.385029 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:57.385003 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccbc8582-ljr7s"] Oct 30 20:28:57.461602 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:57.461511 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxpwc\" (UniqueName: \"kubernetes.io/projected/65afac6e-94e2-4531-aba0-0355ec17aeb7-kube-api-access-fxpwc\") pod \"maas-api-revoke-sub-ccbc8582-ljr7s\" (UID: \"65afac6e-94e2-4531-aba0-0355ec17aeb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccbc8582-ljr7s" Oct 30 20:28:57.461602 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:57.461562 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65afac6e-94e2-4531-aba0-0355ec17aeb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ccbc8582-ljr7s\" (UID: \"65afac6e-94e2-4531-aba0-0355ec17aeb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccbc8582-ljr7s" Oct 30 20:28:57.462213 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:57.461617 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65afac6e-94e2-4531-aba0-0355ec17aeb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ccbc8582-ljr7s\" (UID: \"65afac6e-94e2-4531-aba0-0355ec17aeb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccbc8582-ljr7s" Oct 30 20:28:57.462327 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:57.462307 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65afac6e-94e2-4531-aba0-0355ec17aeb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ccbc8582-ljr7s\" (UID: \"65afac6e-94e2-4531-aba0-0355ec17aeb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccbc8582-ljr7s" Oct 30 20:28:57.464210 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:57.464183 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65afac6e-94e2-4531-aba0-0355ec17aeb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ccbc8582-ljr7s\" (UID: \"65afac6e-94e2-4531-aba0-0355ec17aeb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccbc8582-ljr7s" Oct 30 20:28:57.526040 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:57.526010 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxpwc\" (UniqueName: \"kubernetes.io/projected/65afac6e-94e2-4531-aba0-0355ec17aeb7-kube-api-access-fxpwc\") pod \"maas-api-revoke-sub-ccbc8582-ljr7s\" (UID: \"65afac6e-94e2-4531-aba0-0355ec17aeb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccbc8582-ljr7s" Oct 30 20:28:57.642911 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:57.642881 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccbc8582-ljr7s" Oct 30 20:28:58.025470 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:58.025446 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccbc8582-ljr7s"] Oct 30 20:28:58.027056 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:28:58.027033 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65afac6e_94e2_4531_aba0_0355ec17aeb7.slice/crio-2881eaad02e1957d9a78dcb83a4d1f3d11ef4fa10e74ea8db05a46fa8c728a3b WatchSource:0}: Error finding container 2881eaad02e1957d9a78dcb83a4d1f3d11ef4fa10e74ea8db05a46fa8c728a3b: Status 404 returned error can't find the container with id 2881eaad02e1957d9a78dcb83a4d1f3d11ef4fa10e74ea8db05a46fa8c728a3b Oct 30 20:28:58.192038 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:58.191956 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccbc8582-ljr7s" event={"ID":"65afac6e-94e2-4531-aba0-0355ec17aeb7","Type":"ContainerStarted","Data":"2881eaad02e1957d9a78dcb83a4d1f3d11ef4fa10e74ea8db05a46fa8c728a3b"} Oct 30 20:28:58.582107 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:58.582072 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71a33844-736e-4eeb-b9b9-989bd68a6db3" path="/var/lib/kubelet/pods/71a33844-736e-4eeb-b9b9-989bd68a6db3/volumes" Oct 30 20:28:59.199056 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:59.198965 4292 generic.go:356] "Generic (PLEG): container finished" podID="65afac6e-94e2-4531-aba0-0355ec17aeb7" containerID="2d88c60b26143ad808580b45258989649da670707c5c28d37964c3a6c2744b8c" exitCode=0 Oct 30 20:28:59.199210 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:59.199050 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccbc8582-ljr7s" event={"ID":"65afac6e-94e2-4531-aba0-0355ec17aeb7","Type":"ContainerDied","Data":"2d88c60b26143ad808580b45258989649da670707c5c28d37964c3a6c2744b8c"} Oct 30 20:28:59.740444 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:59.740401 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9389b9-589ddf7656-4l682"] Oct 30 20:28:59.750844 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:59.749831 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-9389b9-589ddf7656-4l682" Oct 30 20:28:59.800402 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:59.800024 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-792px\" (UniqueName: \"kubernetes.io/projected/ae3a4829-e22b-4f59-a1f8-70ce77384238-kube-api-access-792px\") pod \"payload-processing-e2e-extmod-a-9389b9-589ddf7656-4l682\" (UID: \"ae3a4829-e22b-4f59-a1f8-70ce77384238\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9389b9-589ddf7656-4l682" Oct 30 20:28:59.800402 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:59.800207 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae3a4829-e22b-4f59-a1f8-70ce77384238-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-9389b9-589ddf7656-4l682\" (UID: \"ae3a4829-e22b-4f59-a1f8-70ce77384238\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9389b9-589ddf7656-4l682" Oct 30 20:28:59.809857 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:59.808806 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9389b9-589ddf7656-4l682"] Oct 30 20:28:59.809857 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:59.809165 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-9389b9-dockercfg-9gxp4\"" Oct 30 20:28:59.851964 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:59.851927 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-9389b9\"" Oct 30 20:28:59.900899 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:59.900858 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-792px\" (UniqueName: \"kubernetes.io/projected/ae3a4829-e22b-4f59-a1f8-70ce77384238-kube-api-access-792px\") pod \"payload-processing-e2e-extmod-a-9389b9-589ddf7656-4l682\" (UID: \"ae3a4829-e22b-4f59-a1f8-70ce77384238\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9389b9-589ddf7656-4l682" Oct 30 20:28:59.901102 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:59.901020 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae3a4829-e22b-4f59-a1f8-70ce77384238-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-9389b9-589ddf7656-4l682\" (UID: \"ae3a4829-e22b-4f59-a1f8-70ce77384238\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9389b9-589ddf7656-4l682" Oct 30 20:28:59.901762 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:59.901743 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae3a4829-e22b-4f59-a1f8-70ce77384238-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-9389b9-589ddf7656-4l682\" (UID: \"ae3a4829-e22b-4f59-a1f8-70ce77384238\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9389b9-589ddf7656-4l682" Oct 30 20:28:59.920441 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:59.920327 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9389b9-67dfc7496c-fq2ld"] Oct 30 20:28:59.924283 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:59.924262 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9389b9-67dfc7496c-fq2ld" Oct 30 20:28:59.963945 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:59.963909 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-792px\" (UniqueName: \"kubernetes.io/projected/ae3a4829-e22b-4f59-a1f8-70ce77384238-kube-api-access-792px\") pod \"payload-processing-e2e-extmod-a-9389b9-589ddf7656-4l682\" (UID: \"ae3a4829-e22b-4f59-a1f8-70ce77384238\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9389b9-589ddf7656-4l682" Oct 30 20:28:59.978048 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:28:59.978016 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9389b9-67dfc7496c-fq2ld"] Oct 30 20:29:00.002164 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:00.002084 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrvmr\" (UniqueName: \"kubernetes.io/projected/e13a90a4-daaf-462d-ab6a-7d2683c7bdc0-kube-api-access-rrvmr\") pod \"payload-pre-processing-e2e-extmod-a-9389b9-67dfc7496c-fq2ld\" (UID: \"e13a90a4-daaf-462d-ab6a-7d2683c7bdc0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9389b9-67dfc7496c-fq2ld" Oct 30 20:29:00.002330 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:00.002202 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e13a90a4-daaf-462d-ab6a-7d2683c7bdc0-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-9389b9-67dfc7496c-fq2ld\" (UID: \"e13a90a4-daaf-462d-ab6a-7d2683c7bdc0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9389b9-67dfc7496c-fq2ld" Oct 30 20:29:00.097514 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:00.097487 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-9389b9-589ddf7656-4l682" Oct 30 20:29:00.102782 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:00.102760 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e13a90a4-daaf-462d-ab6a-7d2683c7bdc0-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-9389b9-67dfc7496c-fq2ld\" (UID: \"e13a90a4-daaf-462d-ab6a-7d2683c7bdc0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9389b9-67dfc7496c-fq2ld" Oct 30 20:29:00.102868 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:00.102854 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrvmr\" (UniqueName: \"kubernetes.io/projected/e13a90a4-daaf-462d-ab6a-7d2683c7bdc0-kube-api-access-rrvmr\") pod \"payload-pre-processing-e2e-extmod-a-9389b9-67dfc7496c-fq2ld\" (UID: \"e13a90a4-daaf-462d-ab6a-7d2683c7bdc0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9389b9-67dfc7496c-fq2ld" Oct 30 20:29:00.103482 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:00.103460 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e13a90a4-daaf-462d-ab6a-7d2683c7bdc0-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-9389b9-67dfc7496c-fq2ld\" (UID: \"e13a90a4-daaf-462d-ab6a-7d2683c7bdc0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9389b9-67dfc7496c-fq2ld" Oct 30 20:29:00.165463 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:00.165436 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrvmr\" (UniqueName: \"kubernetes.io/projected/e13a90a4-daaf-462d-ab6a-7d2683c7bdc0-kube-api-access-rrvmr\") pod \"payload-pre-processing-e2e-extmod-a-9389b9-67dfc7496c-fq2ld\" (UID: \"e13a90a4-daaf-462d-ab6a-7d2683c7bdc0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9389b9-67dfc7496c-fq2ld" Oct 30 20:29:00.237806 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:00.237773 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9389b9-67dfc7496c-fq2ld" Oct 30 20:29:00.336624 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:00.336593 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccbc8582-ljr7s" Oct 30 20:29:00.409716 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:00.409686 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65afac6e-94e2-4531-aba0-0355ec17aeb7-kube-api-access-fxpwc\" (UniqueName: \"kubernetes.io/projected/65afac6e-94e2-4531-aba0-0355ec17aeb7-kube-api-access-fxpwc\") pod \"65afac6e-94e2-4531-aba0-0355ec17aeb7\" (UID: \"65afac6e-94e2-4531-aba0-0355ec17aeb7\") " Oct 30 20:29:00.409867 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:00.409821 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65afac6e-94e2-4531-aba0-0355ec17aeb7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65afac6e-94e2-4531-aba0-0355ec17aeb7-maas-api-openshift-ca\") pod \"65afac6e-94e2-4531-aba0-0355ec17aeb7\" (UID: \"65afac6e-94e2-4531-aba0-0355ec17aeb7\") " Oct 30 20:29:00.409867 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:00.409858 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65afac6e-94e2-4531-aba0-0355ec17aeb7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65afac6e-94e2-4531-aba0-0355ec17aeb7-maas-api-xks-ca\") pod \"65afac6e-94e2-4531-aba0-0355ec17aeb7\" (UID: \"65afac6e-94e2-4531-aba0-0355ec17aeb7\") " Oct 30 20:29:00.410192 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:00.410172 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65afac6e-94e2-4531-aba0-0355ec17aeb7-maas-api-openshift-ca" pod "65afac6e-94e2-4531-aba0-0355ec17aeb7" (UID: "65afac6e-94e2-4531-aba0-0355ec17aeb7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:29:00.411914 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:00.411890 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65afac6e-94e2-4531-aba0-0355ec17aeb7-kube-api-access-fxpwc" pod "65afac6e-94e2-4531-aba0-0355ec17aeb7" (UID: "65afac6e-94e2-4531-aba0-0355ec17aeb7"). InnerVolumeSpecName "kube-api-access-fxpwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:29:00.411991 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:00.411956 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65afac6e-94e2-4531-aba0-0355ec17aeb7-maas-api-xks-ca" pod "65afac6e-94e2-4531-aba0-0355ec17aeb7" (UID: "65afac6e-94e2-4531-aba0-0355ec17aeb7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:29:00.472777 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:29:00.472749 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae3a4829_e22b_4f59_a1f8_70ce77384238.slice/crio-4109f09d68145828f64a8866629e3a40c205ff22c627437268315f9ef57683ae WatchSource:0}: Error finding container 4109f09d68145828f64a8866629e3a40c205ff22c627437268315f9ef57683ae: Status 404 returned error can't find the container with id 4109f09d68145828f64a8866629e3a40c205ff22c627437268315f9ef57683ae Oct 30 20:29:00.472911 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:00.472856 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9389b9-589ddf7656-4l682"] Oct 30 20:29:00.510826 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:00.510773 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxpwc\" (UniqueName: \"kubernetes.io/projected/65afac6e-94e2-4531-aba0-0355ec17aeb7-kube-api-access-fxpwc\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:29:00.510826 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:00.510798 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65afac6e-94e2-4531-aba0-0355ec17aeb7-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:29:00.510826 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:00.510809 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65afac6e-94e2-4531-aba0-0355ec17aeb7-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:29:00.622498 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:00.622473 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9389b9-67dfc7496c-fq2ld"] Oct 30 20:29:00.623470 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:29:00.623449 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode13a90a4_daaf_462d_ab6a_7d2683c7bdc0.slice/crio-0f3ee086f8d7db527599df2bb05d54c49c5bb1dbc35a90a9ba7ee6b52858118c WatchSource:0}: Error finding container 0f3ee086f8d7db527599df2bb05d54c49c5bb1dbc35a90a9ba7ee6b52858118c: Status 404 returned error can't find the container with id 0f3ee086f8d7db527599df2bb05d54c49c5bb1dbc35a90a9ba7ee6b52858118c Oct 30 20:29:01.214054 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:01.214015 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9389b9-67dfc7496c-fq2ld" event={"ID":"e13a90a4-daaf-462d-ab6a-7d2683c7bdc0","Type":"ContainerStarted","Data":"afcb7457e5f18a77a1ce153bb87228d93018b7e27c712917b2190453d0f52ca7"} Oct 30 20:29:01.214569 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:01.214088 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9389b9-67dfc7496c-fq2ld" Oct 30 20:29:01.214569 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:01.214112 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9389b9-67dfc7496c-fq2ld" event={"ID":"e13a90a4-daaf-462d-ab6a-7d2683c7bdc0","Type":"ContainerStarted","Data":"0f3ee086f8d7db527599df2bb05d54c49c5bb1dbc35a90a9ba7ee6b52858118c"} Oct 30 20:29:01.219845 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:01.219812 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-9389b9-589ddf7656-4l682" event={"ID":"ae3a4829-e22b-4f59-a1f8-70ce77384238","Type":"ContainerStarted","Data":"f92eae1bf5c2bc8266ac9d9a0933689a56f35c64e626eb6ea7351590251d3b30"} Oct 30 20:29:01.220009 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:01.219877 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-9389b9-589ddf7656-4l682" event={"ID":"ae3a4829-e22b-4f59-a1f8-70ce77384238","Type":"ContainerStarted","Data":"4109f09d68145828f64a8866629e3a40c205ff22c627437268315f9ef57683ae"} Oct 30 20:29:01.220009 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:01.219928 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-9389b9-589ddf7656-4l682" Oct 30 20:29:01.221206 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:01.221186 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccbc8582-ljr7s" Oct 30 20:29:01.221311 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:01.221207 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccbc8582-ljr7s" event={"ID":"65afac6e-94e2-4531-aba0-0355ec17aeb7","Type":"ContainerDied","Data":"2881eaad02e1957d9a78dcb83a4d1f3d11ef4fa10e74ea8db05a46fa8c728a3b"} Oct 30 20:29:01.221311 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:01.221241 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2881eaad02e1957d9a78dcb83a4d1f3d11ef4fa10e74ea8db05a46fa8c728a3b" Oct 30 20:29:01.333873 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:01.333819 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9389b9-67dfc7496c-fq2ld" podStartSLOduration=2.333805239 podStartE2EDuration="2.333805239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:28:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:29:01.333252818 +0000 UTC m=+3844.851568300" watchObservedRunningTime="2026-10-30 20:29:01.333805239 +0000 UTC m=+3844.852120696" Oct 30 20:29:01.374971 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:01.374925 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d"] Oct 30 20:29:01.375935 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:01.375697 4292 cpu_manager.go:414] "containerMap: removing container" podUID="65afac6e-94e2-4531-aba0-0355ec17aeb7" containerName="revoke-keys" Oct 30 20:29:01.375935 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:01.375724 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65afac6e-94e2-4531-aba0-0355ec17aeb7" containerName="revoke-keys" Oct 30 20:29:01.375935 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:01.375877 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="65afac6e-94e2-4531-aba0-0355ec17aeb7" containerName="revoke-keys" Oct 30 20:29:01.382856 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:01.382826 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d" Oct 30 20:29:01.421636 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:01.421591 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnhnd\" (UniqueName: \"kubernetes.io/projected/476ee650-d28b-4580-9eb4-a9f095df7822-kube-api-access-xnhnd\") pod \"e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d\" (UID: \"476ee650-d28b-4580-9eb4-a9f095df7822\") " pod="openshift-ingress/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d" Oct 30 20:29:01.421799 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:01.421725 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/476ee650-d28b-4580-9eb4-a9f095df7822-workload-certs\") pod \"e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d\" (UID: \"476ee650-d28b-4580-9eb4-a9f095df7822\") " pod="openshift-ingress/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d" Oct 30 20:29:01.421799 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:01.421782 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/476ee650-d28b-4580-9eb4-a9f095df7822-istiod-ca-cert\") pod \"e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d\" (UID: \"476ee650-d28b-4580-9eb4-a9f095df7822\") " pod="openshift-ingress/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d" Oct 30 20:29:01.421954 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:01.421933 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/476ee650-d28b-4580-9eb4-a9f095df7822-istio-data\") pod \"e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d\" (UID: \"476ee650-d28b-4580-9eb4-a9f095df7822\") " pod="openshift-ingress/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d" Oct 30 20:29:01.422014 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:01.421999 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/476ee650-d28b-4580-9eb4-a9f095df7822-istio-envoy\") pod \"e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d\" (UID: \"476ee650-d28b-4580-9eb4-a9f095df7822\") " pod="openshift-ingress/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d" Oct 30 20:29:01.422122 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:01.422102 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/476ee650-d28b-4580-9eb4-a9f095df7822-credential-socket\") pod \"e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d\" (UID: \"476ee650-d28b-4580-9eb4-a9f095df7822\") " pod="openshift-ingress/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d" Oct 30 20:29:01.422182 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:01.422140 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/476ee650-d28b-4580-9eb4-a9f095df7822-istio-podinfo\") pod \"e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d\" (UID: \"476ee650-d28b-4580-9eb4-a9f095df7822\") " pod="openshift-ingress/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d" Oct 30 20:29:01.422224 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:01.422209 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/476ee650-d28b-4580-9eb4-a9f095df7822-istio-token\") pod \"e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d\" (UID: \"476ee650-d28b-4580-9eb4-a9f095df7822\") " pod="openshift-ingress/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d" Oct 30 20:29:01.422271 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:01.422256 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/476ee650-d28b-4580-9eb4-a9f095df7822-workload-socket\") pod \"e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d\" (UID: \"476ee650-d28b-4580-9eb4-a9f095df7822\") " pod="openshift-ingress/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d" Oct 30 20:29:01.437785 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:01.437738 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-8d612c-openshift-default-dockercfg-j4chs\"" Oct 30 20:29:01.445985 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:01.445955 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d"] Oct 30 20:29:01.517618 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:01.517559 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-9389b9-589ddf7656-4l682" podStartSLOduration=2.517540249 podStartE2EDuration="2.517540249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:28:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:29:01.516865776 +0000 UTC m=+3845.035181241" watchObservedRunningTime="2026-10-30 20:29:01.517540249 +0000 UTC m=+3845.035855707" Oct 30 20:29:01.523077 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:01.523042 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnhnd\" (UniqueName: \"kubernetes.io/projected/476ee650-d28b-4580-9eb4-a9f095df7822-kube-api-access-xnhnd\") pod \"e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d\" (UID: \"476ee650-d28b-4580-9eb4-a9f095df7822\") " pod="openshift-ingress/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d" Oct 30 20:29:01.523249 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:01.523108 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/476ee650-d28b-4580-9eb4-a9f095df7822-workload-certs\") pod \"e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d\" (UID: \"476ee650-d28b-4580-9eb4-a9f095df7822\") " pod="openshift-ingress/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d" Oct 30 20:29:01.523249 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:01.523134 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/476ee650-d28b-4580-9eb4-a9f095df7822-istiod-ca-cert\") pod \"e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d\" (UID: \"476ee650-d28b-4580-9eb4-a9f095df7822\") " pod="openshift-ingress/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d" Oct 30 20:29:01.523249 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:01.523193 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/476ee650-d28b-4580-9eb4-a9f095df7822-istio-data\") pod \"e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d\" (UID: \"476ee650-d28b-4580-9eb4-a9f095df7822\") " pod="openshift-ingress/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d" Oct 30 20:29:01.523468 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:01.523255 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/476ee650-d28b-4580-9eb4-a9f095df7822-istio-envoy\") pod \"e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d\" (UID: \"476ee650-d28b-4580-9eb4-a9f095df7822\") " pod="openshift-ingress/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d" Oct 30 20:29:01.523468 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:01.523295 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/476ee650-d28b-4580-9eb4-a9f095df7822-credential-socket\") pod \"e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d\" (UID: \"476ee650-d28b-4580-9eb4-a9f095df7822\") " pod="openshift-ingress/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d" Oct 30 20:29:01.523468 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:01.523325 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/476ee650-d28b-4580-9eb4-a9f095df7822-istio-podinfo\") pod \"e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d\" (UID: \"476ee650-d28b-4580-9eb4-a9f095df7822\") " pod="openshift-ingress/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d" Oct 30 20:29:01.523468 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:01.523371 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/476ee650-d28b-4580-9eb4-a9f095df7822-istio-token\") pod \"e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d\" (UID: \"476ee650-d28b-4580-9eb4-a9f095df7822\") " pod="openshift-ingress/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d" Oct 30 20:29:01.523468 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:01.523415 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/476ee650-d28b-4580-9eb4-a9f095df7822-workload-socket\") pod \"e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d\" (UID: \"476ee650-d28b-4580-9eb4-a9f095df7822\") " pod="openshift-ingress/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d" Oct 30 20:29:01.523734 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:01.523584 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/476ee650-d28b-4580-9eb4-a9f095df7822-workload-certs\") pod \"e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d\" (UID: \"476ee650-d28b-4580-9eb4-a9f095df7822\") " pod="openshift-ingress/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d" Oct 30 20:29:01.523774 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:01.523758 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/476ee650-d28b-4580-9eb4-a9f095df7822-workload-socket\") pod \"e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d\" (UID: \"476ee650-d28b-4580-9eb4-a9f095df7822\") " pod="openshift-ingress/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d" Oct 30 20:29:01.523875 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:01.523839 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/476ee650-d28b-4580-9eb4-a9f095df7822-istio-data\") pod \"e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d\" (UID: \"476ee650-d28b-4580-9eb4-a9f095df7822\") " pod="openshift-ingress/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d" Oct 30 20:29:01.524831 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:01.524805 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/476ee650-d28b-4580-9eb4-a9f095df7822-istiod-ca-cert\") pod \"e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d\" (UID: \"476ee650-d28b-4580-9eb4-a9f095df7822\") " pod="openshift-ingress/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d" Oct 30 20:29:01.524994 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:01.524910 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/476ee650-d28b-4580-9eb4-a9f095df7822-credential-socket\") pod \"e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d\" (UID: \"476ee650-d28b-4580-9eb4-a9f095df7822\") " pod="openshift-ingress/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d" Oct 30 20:29:01.525817 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:01.525797 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/476ee650-d28b-4580-9eb4-a9f095df7822-istio-envoy\") pod \"e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d\" (UID: \"476ee650-d28b-4580-9eb4-a9f095df7822\") " pod="openshift-ingress/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d" Oct 30 20:29:01.526574 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:01.526554 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/476ee650-d28b-4580-9eb4-a9f095df7822-istio-podinfo\") pod \"e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d\" (UID: \"476ee650-d28b-4580-9eb4-a9f095df7822\") " pod="openshift-ingress/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d" Oct 30 20:29:01.587172 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:01.587075 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/476ee650-d28b-4580-9eb4-a9f095df7822-istio-token\") pod \"e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d\" (UID: \"476ee650-d28b-4580-9eb4-a9f095df7822\") " pod="openshift-ingress/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d" Oct 30 20:29:01.587172 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:01.587090 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnhnd\" (UniqueName: \"kubernetes.io/projected/476ee650-d28b-4580-9eb4-a9f095df7822-kube-api-access-xnhnd\") pod \"e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d\" (UID: \"476ee650-d28b-4580-9eb4-a9f095df7822\") " pod="openshift-ingress/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d" Oct 30 20:29:01.712372 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:01.706373 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d" Oct 30 20:29:02.214984 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:02.214954 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d"] Oct 30 20:29:02.216104 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:29:02.216076 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod476ee650_d28b_4580_9eb4_a9f095df7822.slice/crio-2bc3ed5657fa9701ba5bf52d7549ec20aec86a8209a450e62d5787525ad8a2bb WatchSource:0}: Error finding container 2bc3ed5657fa9701ba5bf52d7549ec20aec86a8209a450e62d5787525ad8a2bb: Status 404 returned error can't find the container with id 2bc3ed5657fa9701ba5bf52d7549ec20aec86a8209a450e62d5787525ad8a2bb Oct 30 20:29:02.219011 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:02.218974 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:29:02.219111 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:02.219062 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:29:02.219154 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:02.219113 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:29:02.230256 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:02.230222 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d" event={"ID":"476ee650-d28b-4580-9eb4-a9f095df7822","Type":"ContainerStarted","Data":"2bc3ed5657fa9701ba5bf52d7549ec20aec86a8209a450e62d5787525ad8a2bb"} Oct 30 20:29:03.236670 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:03.236636 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d" event={"ID":"476ee650-d28b-4580-9eb4-a9f095df7822","Type":"ContainerStarted","Data":"dd97bc70b890574bee4c8fbf27e6c305188815e9e0c39c0e353c371201c010bc"} Oct 30 20:29:03.391589 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:03.391539 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d" podStartSLOduration=2.391524028 podStartE2EDuration="2.391524028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:29:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:29:03.390646931 +0000 UTC m=+3846.908962436" watchObservedRunningTime="2026-10-30 20:29:03.391524028 +0000 UTC m=+3846.909839543" Oct 30 20:29:03.707523 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:03.707488 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d" Oct 30 20:29:03.713602 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:03.713572 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d" Oct 30 20:29:04.242573 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:04.242540 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d" Oct 30 20:29:04.243686 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:04.243665 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d" Oct 30 20:29:08.175466 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:08.175426 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b52b4147-gmggr"] Oct 30 20:29:08.179200 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:08.179177 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b52b4147-gmggr" Oct 30 20:29:08.235206 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:08.235176 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b52b4147-gmggr"] Oct 30 20:29:08.282069 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:08.282033 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:29:08.282069 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:08.282039 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\"" Oct 30 20:29:08.316107 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:08.316071 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35366a1c-b8dc-40e6-b592-296d351702b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b52b4147-gmggr\" (UID: \"35366a1c-b8dc-40e6-b592-296d351702b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b52b4147-gmggr" Oct 30 20:29:08.316290 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:08.316214 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svkq2\" (UniqueName: \"kubernetes.io/projected/35366a1c-b8dc-40e6-b592-296d351702b5-kube-api-access-svkq2\") pod \"maas-api-revoke-sub-b52b4147-gmggr\" (UID: \"35366a1c-b8dc-40e6-b592-296d351702b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b52b4147-gmggr" Oct 30 20:29:08.316290 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:08.316259 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35366a1c-b8dc-40e6-b592-296d351702b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b52b4147-gmggr\" (UID: \"35366a1c-b8dc-40e6-b592-296d351702b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b52b4147-gmggr" Oct 30 20:29:08.422213 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:08.420252 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35366a1c-b8dc-40e6-b592-296d351702b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b52b4147-gmggr\" (UID: \"35366a1c-b8dc-40e6-b592-296d351702b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b52b4147-gmggr" Oct 30 20:29:08.422213 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:08.420466 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35366a1c-b8dc-40e6-b592-296d351702b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b52b4147-gmggr\" (UID: \"35366a1c-b8dc-40e6-b592-296d351702b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b52b4147-gmggr" Oct 30 20:29:08.422213 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:08.420595 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svkq2\" (UniqueName: \"kubernetes.io/projected/35366a1c-b8dc-40e6-b592-296d351702b5-kube-api-access-svkq2\") pod \"maas-api-revoke-sub-b52b4147-gmggr\" (UID: \"35366a1c-b8dc-40e6-b592-296d351702b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b52b4147-gmggr" Oct 30 20:29:08.422213 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:08.421782 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35366a1c-b8dc-40e6-b592-296d351702b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b52b4147-gmggr\" (UID: \"35366a1c-b8dc-40e6-b592-296d351702b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b52b4147-gmggr" Oct 30 20:29:08.431632 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:08.431559 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35366a1c-b8dc-40e6-b592-296d351702b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b52b4147-gmggr\" (UID: \"35366a1c-b8dc-40e6-b592-296d351702b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b52b4147-gmggr" Oct 30 20:29:08.492852 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:08.492822 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svkq2\" (UniqueName: \"kubernetes.io/projected/35366a1c-b8dc-40e6-b592-296d351702b5-kube-api-access-svkq2\") pod \"maas-api-revoke-sub-b52b4147-gmggr\" (UID: \"35366a1c-b8dc-40e6-b592-296d351702b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b52b4147-gmggr" Oct 30 20:29:08.494443 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:08.494421 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b52b4147-gmggr" Oct 30 20:29:08.809070 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:08.809030 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-cr258"] Oct 30 20:29:08.810641 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:08.810605 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-cr258" podUID="a0f64633-5da3-4c5d-992d-e6254a825c85" containerName="istio-proxy" containerID="cri-o://905bd0a8aed4d3fef2a848e2fd395486f034de41443b7217b230c84a5dec3d02" gracePeriod=30 Oct 30 20:29:08.941254 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:08.941191 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b52b4147-gmggr"] Oct 30 20:29:08.942015 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:29:08.941988 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35366a1c_b8dc_40e6_b592_296d351702b5.slice/crio-60835c475e5c9a0a76a012e8cab89dcb3077e4932d2fe69e296a18a04208d2eb WatchSource:0}: Error finding container 60835c475e5c9a0a76a012e8cab89dcb3077e4932d2fe69e296a18a04208d2eb: Status 404 returned error can't find the container with id 60835c475e5c9a0a76a012e8cab89dcb3077e4932d2fe69e296a18a04208d2eb Oct 30 20:29:09.270959 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:09.270924 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b52b4147-gmggr" event={"ID":"35366a1c-b8dc-40e6-b592-296d351702b5","Type":"ContainerStarted","Data":"53420a01c5c04acfe36b84d7e2079e3d7cbe40037517b42b922582e6e2982b7f"} Oct 30 20:29:09.271365 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:09.270992 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b52b4147-gmggr" event={"ID":"35366a1c-b8dc-40e6-b592-296d351702b5","Type":"ContainerStarted","Data":"60835c475e5c9a0a76a012e8cab89dcb3077e4932d2fe69e296a18a04208d2eb"} Oct 30 20:29:09.391442 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:09.391376 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b52b4147-gmggr" podStartSLOduration=1.391342425 podStartE2EDuration="1.391342425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:29:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:29:09.38918667 +0000 UTC m=+3852.907502127" watchObservedRunningTime="2026-10-30 20:29:09.391342425 +0000 UTC m=+3852.909657931" Oct 30 20:29:10.278066 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:10.278032 4292 generic.go:356] "Generic (PLEG): container finished" podID="35366a1c-b8dc-40e6-b592-296d351702b5" containerID="53420a01c5c04acfe36b84d7e2079e3d7cbe40037517b42b922582e6e2982b7f" exitCode=0 Oct 30 20:29:10.278472 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:10.278118 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b52b4147-gmggr" event={"ID":"35366a1c-b8dc-40e6-b592-296d351702b5","Type":"ContainerDied","Data":"53420a01c5c04acfe36b84d7e2079e3d7cbe40037517b42b922582e6e2982b7f"} Oct 30 20:29:11.338574 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:11.338525 4292 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-cr258 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:29:11.338993 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:11.338596 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-cr258" podUID="a0f64633-5da3-4c5d-992d-e6254a825c85" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:29:11.416658 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:11.416629 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b52b4147-gmggr" Oct 30 20:29:11.598262 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:11.594404 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35366a1c-b8dc-40e6-b592-296d351702b5-kube-api-access-svkq2\" (UniqueName: \"kubernetes.io/projected/35366a1c-b8dc-40e6-b592-296d351702b5-kube-api-access-svkq2\") pod \"35366a1c-b8dc-40e6-b592-296d351702b5\" (UID: \"35366a1c-b8dc-40e6-b592-296d351702b5\") " Oct 30 20:29:11.598262 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:11.594540 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35366a1c-b8dc-40e6-b592-296d351702b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35366a1c-b8dc-40e6-b592-296d351702b5-maas-api-openshift-ca\") pod \"35366a1c-b8dc-40e6-b592-296d351702b5\" (UID: \"35366a1c-b8dc-40e6-b592-296d351702b5\") " Oct 30 20:29:11.598262 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:11.594570 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35366a1c-b8dc-40e6-b592-296d351702b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35366a1c-b8dc-40e6-b592-296d351702b5-maas-api-xks-ca\") pod \"35366a1c-b8dc-40e6-b592-296d351702b5\" (UID: \"35366a1c-b8dc-40e6-b592-296d351702b5\") " Oct 30 20:29:11.598262 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:11.595713 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35366a1c-b8dc-40e6-b592-296d351702b5-maas-api-openshift-ca" pod "35366a1c-b8dc-40e6-b592-296d351702b5" (UID: "35366a1c-b8dc-40e6-b592-296d351702b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:29:11.599103 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:11.599073 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35366a1c-b8dc-40e6-b592-296d351702b5-kube-api-access-svkq2" pod "35366a1c-b8dc-40e6-b592-296d351702b5" (UID: "35366a1c-b8dc-40e6-b592-296d351702b5"). InnerVolumeSpecName "kube-api-access-svkq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:29:11.603083 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:11.603056 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35366a1c-b8dc-40e6-b592-296d351702b5-maas-api-xks-ca" pod "35366a1c-b8dc-40e6-b592-296d351702b5" (UID: "35366a1c-b8dc-40e6-b592-296d351702b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:29:11.695705 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:11.695678 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35366a1c-b8dc-40e6-b592-296d351702b5-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:29:11.695705 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:11.695707 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35366a1c-b8dc-40e6-b592-296d351702b5-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:29:11.696027 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:11.695721 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svkq2\" (UniqueName: \"kubernetes.io/projected/35366a1c-b8dc-40e6-b592-296d351702b5-kube-api-access-svkq2\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:29:12.233441 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:12.233411 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9389b9-67dfc7496c-fq2ld" Oct 30 20:29:12.233620 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:12.233487 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-9389b9-589ddf7656-4l682" Oct 30 20:29:12.291510 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:12.291480 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b52b4147-gmggr" Oct 30 20:29:12.291680 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:12.291522 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b52b4147-gmggr" event={"ID":"35366a1c-b8dc-40e6-b592-296d351702b5","Type":"ContainerDied","Data":"60835c475e5c9a0a76a012e8cab89dcb3077e4932d2fe69e296a18a04208d2eb"} Oct 30 20:29:12.291680 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:12.291571 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60835c475e5c9a0a76a012e8cab89dcb3077e4932d2fe69e296a18a04208d2eb" Oct 30 20:29:12.429135 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:12.429107 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8d612c-74c797d987-9mbvg"] Oct 30 20:29:12.429693 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:12.429676 4292 cpu_manager.go:414] "containerMap: removing container" podUID="35366a1c-b8dc-40e6-b592-296d351702b5" containerName="revoke-keys" Oct 30 20:29:12.429760 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:12.429695 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35366a1c-b8dc-40e6-b592-296d351702b5" containerName="revoke-keys" Oct 30 20:29:12.429803 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:12.429790 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="35366a1c-b8dc-40e6-b592-296d351702b5" containerName="revoke-keys" Oct 30 20:29:12.432751 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:12.432729 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8d612c-74c797d987-9mbvg" Oct 30 20:29:12.487074 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:12.486999 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-8d612c\"" Oct 30 20:29:12.487291 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:12.487033 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-8d612c-dockercfg-hrdcp\"" Oct 30 20:29:12.490802 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:12.490778 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8d612c-74c797d987-9mbvg"] Oct 30 20:29:12.503798 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:12.503767 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73fcfe52-4127-40a3-a834-13200b4d2e85-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-8d612c-74c797d987-9mbvg\" (UID: \"73fcfe52-4127-40a3-a834-13200b4d2e85\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8d612c-74c797d987-9mbvg" Oct 30 20:29:12.503911 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:12.503893 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqnzx\" (UniqueName: \"kubernetes.io/projected/73fcfe52-4127-40a3-a834-13200b4d2e85-kube-api-access-mqnzx\") pod \"payload-pre-processing-e2e-extmod-a-8d612c-74c797d987-9mbvg\" (UID: \"73fcfe52-4127-40a3-a834-13200b4d2e85\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8d612c-74c797d987-9mbvg" Oct 30 20:29:12.605449 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:12.605395 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73fcfe52-4127-40a3-a834-13200b4d2e85-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-8d612c-74c797d987-9mbvg\" (UID: \"73fcfe52-4127-40a3-a834-13200b4d2e85\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8d612c-74c797d987-9mbvg" Oct 30 20:29:12.608338 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:12.608210 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73fcfe52-4127-40a3-a834-13200b4d2e85-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-8d612c-74c797d987-9mbvg\" (UID: \"73fcfe52-4127-40a3-a834-13200b4d2e85\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8d612c-74c797d987-9mbvg" Oct 30 20:29:12.608511 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:12.608320 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqnzx\" (UniqueName: \"kubernetes.io/projected/73fcfe52-4127-40a3-a834-13200b4d2e85-kube-api-access-mqnzx\") pod \"payload-pre-processing-e2e-extmod-a-8d612c-74c797d987-9mbvg\" (UID: \"73fcfe52-4127-40a3-a834-13200b4d2e85\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8d612c-74c797d987-9mbvg" Oct 30 20:29:12.706461 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:12.706426 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqnzx\" (UniqueName: \"kubernetes.io/projected/73fcfe52-4127-40a3-a834-13200b4d2e85-kube-api-access-mqnzx\") pod \"payload-pre-processing-e2e-extmod-a-8d612c-74c797d987-9mbvg\" (UID: \"73fcfe52-4127-40a3-a834-13200b4d2e85\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8d612c-74c797d987-9mbvg" Oct 30 20:29:12.746146 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:12.746068 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8d612c-74c797d987-9mbvg" Oct 30 20:29:13.123868 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:13.123846 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8d612c-74c797d987-9mbvg"] Oct 30 20:29:13.125279 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:29:13.125251 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73fcfe52_4127_40a3_a834_13200b4d2e85.slice/crio-bb4c19f9cb5bf3203200ff87ce560b1a6ed19448125cd2f4d562fcc0980b3453 WatchSource:0}: Error finding container bb4c19f9cb5bf3203200ff87ce560b1a6ed19448125cd2f4d562fcc0980b3453: Status 404 returned error can't find the container with id bb4c19f9cb5bf3203200ff87ce560b1a6ed19448125cd2f4d562fcc0980b3453 Oct 30 20:29:13.219845 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:29:13.219814 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:29:13.299068 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:13.299033 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8d612c-74c797d987-9mbvg" event={"ID":"73fcfe52-4127-40a3-a834-13200b4d2e85","Type":"ContainerStarted","Data":"baaf54e57cadce5f904e423993c955e5ae41c4481a5bd8e54b955f9bb8d00ceb"} Oct 30 20:29:13.299205 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:13.299095 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8d612c-74c797d987-9mbvg" Oct 30 20:29:13.299205 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:13.299117 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8d612c-74c797d987-9mbvg" event={"ID":"73fcfe52-4127-40a3-a834-13200b4d2e85","Type":"ContainerStarted","Data":"bb4c19f9cb5bf3203200ff87ce560b1a6ed19448125cd2f4d562fcc0980b3453"} Oct 30 20:29:13.419217 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:13.419124 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8d612c-74c797d987-9mbvg" podStartSLOduration=1.419109628 podStartE2EDuration="1.419109628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:29:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:29:13.417735723 +0000 UTC m=+3856.936051161" watchObservedRunningTime="2026-10-30 20:29:13.419109628 +0000 UTC m=+3856.937425085" Oct 30 20:29:14.066560 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:14.066534 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-cr258" Oct 30 20:29:14.125099 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:14.125012 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0f64633-5da3-4c5d-992d-e6254a825c85-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0f64633-5da3-4c5d-992d-e6254a825c85-istio-envoy\") pod \"a0f64633-5da3-4c5d-992d-e6254a825c85\" (UID: \"a0f64633-5da3-4c5d-992d-e6254a825c85\") " Oct 30 20:29:14.125388 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:14.125346 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0f64633-5da3-4c5d-992d-e6254a825c85-kube-api-access-g5dpk\" (UniqueName: \"kubernetes.io/projected/a0f64633-5da3-4c5d-992d-e6254a825c85-kube-api-access-g5dpk\") pod \"a0f64633-5da3-4c5d-992d-e6254a825c85\" (UID: \"a0f64633-5da3-4c5d-992d-e6254a825c85\") " Oct 30 20:29:14.125523 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:14.125510 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0f64633-5da3-4c5d-992d-e6254a825c85-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0f64633-5da3-4c5d-992d-e6254a825c85-workload-socket\") pod \"a0f64633-5da3-4c5d-992d-e6254a825c85\" (UID: \"a0f64633-5da3-4c5d-992d-e6254a825c85\") " Oct 30 20:29:14.125652 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:14.125640 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a0f64633-5da3-4c5d-992d-e6254a825c85-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0f64633-5da3-4c5d-992d-e6254a825c85-istio-podinfo\") pod \"a0f64633-5da3-4c5d-992d-e6254a825c85\" (UID: \"a0f64633-5da3-4c5d-992d-e6254a825c85\") " Oct 30 20:29:14.125811 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:14.125797 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0f64633-5da3-4c5d-992d-e6254a825c85-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0f64633-5da3-4c5d-992d-e6254a825c85-istiod-ca-cert\") pod \"a0f64633-5da3-4c5d-992d-e6254a825c85\" (UID: \"a0f64633-5da3-4c5d-992d-e6254a825c85\") " Oct 30 20:29:14.125964 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:14.125951 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0f64633-5da3-4c5d-992d-e6254a825c85-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0f64633-5da3-4c5d-992d-e6254a825c85-workload-certs\") pod \"a0f64633-5da3-4c5d-992d-e6254a825c85\" (UID: \"a0f64633-5da3-4c5d-992d-e6254a825c85\") " Oct 30 20:29:14.126098 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:14.126085 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0f64633-5da3-4c5d-992d-e6254a825c85-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0f64633-5da3-4c5d-992d-e6254a825c85-istio-data\") pod \"a0f64633-5da3-4c5d-992d-e6254a825c85\" (UID: \"a0f64633-5da3-4c5d-992d-e6254a825c85\") " Oct 30 20:29:14.126215 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:14.126200 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0f64633-5da3-4c5d-992d-e6254a825c85-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0f64633-5da3-4c5d-992d-e6254a825c85-credential-socket\") pod \"a0f64633-5da3-4c5d-992d-e6254a825c85\" (UID: \"a0f64633-5da3-4c5d-992d-e6254a825c85\") " Oct 30 20:29:14.126333 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:14.126322 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0f64633-5da3-4c5d-992d-e6254a825c85-istio-token\" (UniqueName: \"kubernetes.io/projected/a0f64633-5da3-4c5d-992d-e6254a825c85-istio-token\") pod \"a0f64633-5da3-4c5d-992d-e6254a825c85\" (UID: \"a0f64633-5da3-4c5d-992d-e6254a825c85\") " Oct 30 20:29:14.127222 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:14.127124 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0f64633-5da3-4c5d-992d-e6254a825c85-istiod-ca-cert" pod "a0f64633-5da3-4c5d-992d-e6254a825c85" (UID: "a0f64633-5da3-4c5d-992d-e6254a825c85"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:29:14.127582 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:14.127293 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0f64633-5da3-4c5d-992d-e6254a825c85-workload-certs" pod "a0f64633-5da3-4c5d-992d-e6254a825c85" (UID: "a0f64633-5da3-4c5d-992d-e6254a825c85"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:29:14.127582 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:14.127312 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0f64633-5da3-4c5d-992d-e6254a825c85-workload-socket" pod "a0f64633-5da3-4c5d-992d-e6254a825c85" (UID: "a0f64633-5da3-4c5d-992d-e6254a825c85"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:29:14.127748 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:14.127452 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0f64633-5da3-4c5d-992d-e6254a825c85-istio-data" pod "a0f64633-5da3-4c5d-992d-e6254a825c85" (UID: "a0f64633-5da3-4c5d-992d-e6254a825c85"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:29:14.127814 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:14.127773 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0f64633-5da3-4c5d-992d-e6254a825c85-workload-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:29:14.127814 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:14.127794 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0f64633-5da3-4c5d-992d-e6254a825c85-istio-data\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:29:14.127814 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:14.127808 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0f64633-5da3-4c5d-992d-e6254a825c85-workload-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:29:14.127987 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:14.127819 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0f64633-5da3-4c5d-992d-e6254a825c85-istiod-ca-cert\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:29:14.128450 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:14.128424 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0f64633-5da3-4c5d-992d-e6254a825c85-credential-socket" pod "a0f64633-5da3-4c5d-992d-e6254a825c85" (UID: "a0f64633-5da3-4c5d-992d-e6254a825c85"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:29:14.129580 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:14.129555 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0f64633-5da3-4c5d-992d-e6254a825c85-istio-token" pod "a0f64633-5da3-4c5d-992d-e6254a825c85" (UID: "a0f64633-5da3-4c5d-992d-e6254a825c85"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:29:14.129656 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:14.129574 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0f64633-5da3-4c5d-992d-e6254a825c85-kube-api-access-g5dpk" pod "a0f64633-5da3-4c5d-992d-e6254a825c85" (UID: "a0f64633-5da3-4c5d-992d-e6254a825c85"). InnerVolumeSpecName "kube-api-access-g5dpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:29:14.129839 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:14.129815 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a0f64633-5da3-4c5d-992d-e6254a825c85-istio-podinfo" pod "a0f64633-5da3-4c5d-992d-e6254a825c85" (UID: "a0f64633-5da3-4c5d-992d-e6254a825c85"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:29:14.129839 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:14.129813 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0f64633-5da3-4c5d-992d-e6254a825c85-istio-envoy" pod "a0f64633-5da3-4c5d-992d-e6254a825c85" (UID: "a0f64633-5da3-4c5d-992d-e6254a825c85"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:29:14.228848 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:14.228805 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0f64633-5da3-4c5d-992d-e6254a825c85-credential-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:29:14.228848 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:14.228846 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0f64633-5da3-4c5d-992d-e6254a825c85-istio-token\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:29:14.229046 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:14.228861 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0f64633-5da3-4c5d-992d-e6254a825c85-istio-envoy\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:29:14.229046 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:14.228876 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5dpk\" (UniqueName: \"kubernetes.io/projected/a0f64633-5da3-4c5d-992d-e6254a825c85-kube-api-access-g5dpk\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:29:14.229046 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:14.228894 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0f64633-5da3-4c5d-992d-e6254a825c85-istio-podinfo\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:29:14.306253 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:14.306223 4292 generic.go:356] "Generic (PLEG): container finished" podID="a0f64633-5da3-4c5d-992d-e6254a825c85" containerID="905bd0a8aed4d3fef2a848e2fd395486f034de41443b7217b230c84a5dec3d02" exitCode=0 Oct 30 20:29:14.306443 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:14.306285 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-cr258" Oct 30 20:29:14.306443 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:14.306307 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-cr258" event={"ID":"a0f64633-5da3-4c5d-992d-e6254a825c85","Type":"ContainerDied","Data":"905bd0a8aed4d3fef2a848e2fd395486f034de41443b7217b230c84a5dec3d02"} Oct 30 20:29:14.306443 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:14.306407 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-cr258" event={"ID":"a0f64633-5da3-4c5d-992d-e6254a825c85","Type":"ContainerDied","Data":"0ec3fdd349e5c90098ab38af4d7a4890c4f4682d3ff6bded72c6c6dba7fde3fc"} Oct 30 20:29:14.306443 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:14.306432 4292 scope.go:122] "RemoveContainer" containerID="905bd0a8aed4d3fef2a848e2fd395486f034de41443b7217b230c84a5dec3d02" Oct 30 20:29:14.321577 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:14.321556 4292 scope.go:122] "RemoveContainer" containerID="905bd0a8aed4d3fef2a848e2fd395486f034de41443b7217b230c84a5dec3d02" Oct 30 20:29:14.321898 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:29:14.321879 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"905bd0a8aed4d3fef2a848e2fd395486f034de41443b7217b230c84a5dec3d02\": container with ID starting with 905bd0a8aed4d3fef2a848e2fd395486f034de41443b7217b230c84a5dec3d02 not found: ID does not exist" containerID="905bd0a8aed4d3fef2a848e2fd395486f034de41443b7217b230c84a5dec3d02" Oct 30 20:29:14.321943 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:14.321909 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"905bd0a8aed4d3fef2a848e2fd395486f034de41443b7217b230c84a5dec3d02"} err="failed to get container status \"905bd0a8aed4d3fef2a848e2fd395486f034de41443b7217b230c84a5dec3d02\": rpc error: code = NotFound desc = could not find container \"905bd0a8aed4d3fef2a848e2fd395486f034de41443b7217b230c84a5dec3d02\": container with ID starting with 905bd0a8aed4d3fef2a848e2fd395486f034de41443b7217b230c84a5dec3d02 not found: ID does not exist" Oct 30 20:29:14.484241 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:14.484165 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-cr258"] Oct 30 20:29:14.494683 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:14.494656 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-cr258"] Oct 30 20:29:14.582324 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:14.582294 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0f64633-5da3-4c5d-992d-e6254a825c85" path="/var/lib/kubelet/pods/a0f64633-5da3-4c5d-992d-e6254a825c85/volumes" Oct 30 20:29:17.470046 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:17.470009 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b91df1-5c7bbf65f-vvg2l"] Oct 30 20:29:17.470495 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:17.470481 4292 cpu_manager.go:414] "containerMap: removing container" podUID="a0f64633-5da3-4c5d-992d-e6254a825c85" containerName="istio-proxy" Oct 30 20:29:17.470554 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:17.470495 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0f64633-5da3-4c5d-992d-e6254a825c85" containerName="istio-proxy" Oct 30 20:29:17.470587 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:17.470580 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0f64633-5da3-4c5d-992d-e6254a825c85" containerName="istio-proxy" Oct 30 20:29:17.475497 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:17.475476 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b91df1-5c7bbf65f-vvg2l" Oct 30 20:29:17.534169 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:17.534137 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b91df1-5c7bbf65f-vvg2l"] Oct 30 20:29:17.592371 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:17.588600 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-b91df1-dockercfg-dcb9j\"" Oct 30 20:29:17.592371 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:17.588703 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-b91df1\"" Oct 30 20:29:17.596829 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:17.596797 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-b91df1-765b88fb56-l2xpb"] Oct 30 20:29:17.601899 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:17.601875 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b91df1-765b88fb56-l2xpb" Oct 30 20:29:17.610099 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:17.608895 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b07a622c-18d8-4351-a871-6b019ba2a2a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-b91df1-5c7bbf65f-vvg2l\" (UID: \"b07a622c-18d8-4351-a871-6b019ba2a2a4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b91df1-5c7bbf65f-vvg2l" Oct 30 20:29:17.610099 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:17.609108 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c675v\" (UniqueName: \"kubernetes.io/projected/b07a622c-18d8-4351-a871-6b019ba2a2a4-kube-api-access-c675v\") pod \"payload-pre-processing-e2e-shared-b-w4-b91df1-5c7bbf65f-vvg2l\" (UID: \"b07a622c-18d8-4351-a871-6b019ba2a2a4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b91df1-5c7bbf65f-vvg2l" Oct 30 20:29:17.657254 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:17.657221 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-b91df1-765b88fb56-l2xpb"] Oct 30 20:29:17.710145 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:17.710112 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2x75\" (UniqueName: \"kubernetes.io/projected/335d81c9-5d05-42b9-8d9b-90974d402605-kube-api-access-d2x75\") pod \"payload-processing-e2e-shared-b-w4-b91df1-765b88fb56-l2xpb\" (UID: \"335d81c9-5d05-42b9-8d9b-90974d402605\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b91df1-765b88fb56-l2xpb" Oct 30 20:29:17.710307 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:17.710155 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c675v\" (UniqueName: \"kubernetes.io/projected/b07a622c-18d8-4351-a871-6b019ba2a2a4-kube-api-access-c675v\") pod \"payload-pre-processing-e2e-shared-b-w4-b91df1-5c7bbf65f-vvg2l\" (UID: \"b07a622c-18d8-4351-a871-6b019ba2a2a4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b91df1-5c7bbf65f-vvg2l" Oct 30 20:29:17.710374 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:17.710311 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b07a622c-18d8-4351-a871-6b019ba2a2a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-b91df1-5c7bbf65f-vvg2l\" (UID: \"b07a622c-18d8-4351-a871-6b019ba2a2a4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b91df1-5c7bbf65f-vvg2l" Oct 30 20:29:17.710425 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:17.710394 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/335d81c9-5d05-42b9-8d9b-90974d402605-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-b91df1-765b88fb56-l2xpb\" (UID: \"335d81c9-5d05-42b9-8d9b-90974d402605\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b91df1-765b88fb56-l2xpb" Oct 30 20:29:17.710990 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:17.710972 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b07a622c-18d8-4351-a871-6b019ba2a2a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-b91df1-5c7bbf65f-vvg2l\" (UID: \"b07a622c-18d8-4351-a871-6b019ba2a2a4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b91df1-5c7bbf65f-vvg2l" Oct 30 20:29:17.772975 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:17.772944 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c675v\" (UniqueName: \"kubernetes.io/projected/b07a622c-18d8-4351-a871-6b019ba2a2a4-kube-api-access-c675v\") pod \"payload-pre-processing-e2e-shared-b-w4-b91df1-5c7bbf65f-vvg2l\" (UID: \"b07a622c-18d8-4351-a871-6b019ba2a2a4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b91df1-5c7bbf65f-vvg2l" Oct 30 20:29:17.788775 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:17.788749 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b91df1-5c7bbf65f-vvg2l" Oct 30 20:29:17.812069 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:17.812038 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/335d81c9-5d05-42b9-8d9b-90974d402605-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-b91df1-765b88fb56-l2xpb\" (UID: \"335d81c9-5d05-42b9-8d9b-90974d402605\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b91df1-765b88fb56-l2xpb" Oct 30 20:29:17.812207 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:17.812192 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2x75\" (UniqueName: \"kubernetes.io/projected/335d81c9-5d05-42b9-8d9b-90974d402605-kube-api-access-d2x75\") pod \"payload-processing-e2e-shared-b-w4-b91df1-765b88fb56-l2xpb\" (UID: \"335d81c9-5d05-42b9-8d9b-90974d402605\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b91df1-765b88fb56-l2xpb" Oct 30 20:29:17.812865 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:17.812839 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/335d81c9-5d05-42b9-8d9b-90974d402605-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-b91df1-765b88fb56-l2xpb\" (UID: \"335d81c9-5d05-42b9-8d9b-90974d402605\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b91df1-765b88fb56-l2xpb" Oct 30 20:29:17.876983 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:17.876955 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2x75\" (UniqueName: \"kubernetes.io/projected/335d81c9-5d05-42b9-8d9b-90974d402605-kube-api-access-d2x75\") pod \"payload-processing-e2e-shared-b-w4-b91df1-765b88fb56-l2xpb\" (UID: \"335d81c9-5d05-42b9-8d9b-90974d402605\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b91df1-765b88fb56-l2xpb" Oct 30 20:29:17.941050 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:17.941017 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b91df1-765b88fb56-l2xpb" Oct 30 20:29:18.165491 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:18.165466 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b91df1-5c7bbf65f-vvg2l"] Oct 30 20:29:18.166737 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:29:18.166708 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb07a622c_18d8_4351_a871_6b019ba2a2a4.slice/crio-ced609a98cd9adf084a5f570f59c4e927914cc0565935567967f6ce5bad30f16 WatchSource:0}: Error finding container ced609a98cd9adf084a5f570f59c4e927914cc0565935567967f6ce5bad30f16: Status 404 returned error can't find the container with id ced609a98cd9adf084a5f570f59c4e927914cc0565935567967f6ce5bad30f16 Oct 30 20:29:18.318860 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:18.318834 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-b91df1-765b88fb56-l2xpb"] Oct 30 20:29:18.320083 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:29:18.320058 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod335d81c9_5d05_42b9_8d9b_90974d402605.slice/crio-bcc298ab18fbf32e7e964f9a6f0fe7889007b332353ef8a0635175edc9f73781 WatchSource:0}: Error finding container bcc298ab18fbf32e7e964f9a6f0fe7889007b332353ef8a0635175edc9f73781: Status 404 returned error can't find the container with id bcc298ab18fbf32e7e964f9a6f0fe7889007b332353ef8a0635175edc9f73781 Oct 30 20:29:18.342664 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:18.342637 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b91df1-765b88fb56-l2xpb" event={"ID":"335d81c9-5d05-42b9-8d9b-90974d402605","Type":"ContainerStarted","Data":"bcc298ab18fbf32e7e964f9a6f0fe7889007b332353ef8a0635175edc9f73781"} Oct 30 20:29:18.344190 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:18.344166 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b91df1-5c7bbf65f-vvg2l" event={"ID":"b07a622c-18d8-4351-a871-6b019ba2a2a4","Type":"ContainerStarted","Data":"f0e544a3c259b6d8e9fae717944a1827849036674765c5a974b8ea30674d13e0"} Oct 30 20:29:18.344302 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:18.344208 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b91df1-5c7bbf65f-vvg2l" event={"ID":"b07a622c-18d8-4351-a871-6b019ba2a2a4","Type":"ContainerStarted","Data":"ced609a98cd9adf084a5f570f59c4e927914cc0565935567967f6ce5bad30f16"} Oct 30 20:29:18.344302 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:18.344231 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b91df1-5c7bbf65f-vvg2l" Oct 30 20:29:18.462281 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:18.462195 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b91df1-5c7bbf65f-vvg2l" podStartSLOduration=1.462180797 podStartE2EDuration="1.462180797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:29:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:29:18.461779777 +0000 UTC m=+3861.980095304" watchObservedRunningTime="2026-10-30 20:29:18.462180797 +0000 UTC m=+3861.980496255" Oct 30 20:29:19.351449 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:19.351408 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b91df1-765b88fb56-l2xpb" event={"ID":"335d81c9-5d05-42b9-8d9b-90974d402605","Type":"ContainerStarted","Data":"14a296991022fb52878c2eea4e125e7e0b21f241ea5827fd9d3578376bcbdbc1"} Oct 30 20:29:19.351893 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:19.351597 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b91df1-765b88fb56-l2xpb" Oct 30 20:29:19.472032 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:19.471966 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b91df1-765b88fb56-l2xpb" podStartSLOduration=2.471950894 podStartE2EDuration="2.471950894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:29:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:29:19.469528404 +0000 UTC m=+3862.987843860" watchObservedRunningTime="2026-10-30 20:29:19.471950894 +0000 UTC m=+3862.990266351" Oct 30 20:29:24.309750 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:24.309717 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8d612c-74c797d987-9mbvg" Oct 30 20:29:29.354663 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:29.354631 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b91df1-5c7bbf65f-vvg2l" Oct 30 20:29:30.360494 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:30.360462 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b91df1-765b88fb56-l2xpb" Oct 30 20:29:34.449731 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:34.449691 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-056a27/auth-test-model-056a27-kserve-bdf98c7c-k7tgb"] Oct 30 20:29:43.220961 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:29:43.220924 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:29:49.469819 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:49.469770 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-056a27/auth-test-model-056a27-kserve-bdf98c7c-k7tgb" podUID="c29e722c-2f9e-4f04-acc4-86cc0d364373" containerName="main" containerID="cri-o://ff9e5164fd7e5c208041fe711c5e99fa6384e3393ea3ca3ca45afaf4ba19ff7f" gracePeriod=45 Oct 30 20:29:49.731485 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:49.731430 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-056a27/auth-test-model-056a27-kserve-bdf98c7c-k7tgb" Oct 30 20:29:49.821969 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:49.821940 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c29e722c-2f9e-4f04-acc4-86cc0d364373-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c29e722c-2f9e-4f04-acc4-86cc0d364373-tmp-dir\") pod \"c29e722c-2f9e-4f04-acc4-86cc0d364373\" (UID: \"c29e722c-2f9e-4f04-acc4-86cc0d364373\") " Oct 30 20:29:49.821969 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:49.821973 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c29e722c-2f9e-4f04-acc4-86cc0d364373-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c29e722c-2f9e-4f04-acc4-86cc0d364373-model-cache\") pod \"c29e722c-2f9e-4f04-acc4-86cc0d364373\" (UID: \"c29e722c-2f9e-4f04-acc4-86cc0d364373\") " Oct 30 20:29:49.822170 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:49.822024 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c29e722c-2f9e-4f04-acc4-86cc0d364373-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c29e722c-2f9e-4f04-acc4-86cc0d364373-dshm\") pod \"c29e722c-2f9e-4f04-acc4-86cc0d364373\" (UID: \"c29e722c-2f9e-4f04-acc4-86cc0d364373\") " Oct 30 20:29:49.822170 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:49.822067 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c29e722c-2f9e-4f04-acc4-86cc0d364373-home\" (UniqueName: \"kubernetes.io/empty-dir/c29e722c-2f9e-4f04-acc4-86cc0d364373-home\") pod \"c29e722c-2f9e-4f04-acc4-86cc0d364373\" (UID: \"c29e722c-2f9e-4f04-acc4-86cc0d364373\") " Oct 30 20:29:49.822170 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:49.822103 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c29e722c-2f9e-4f04-acc4-86cc0d364373-kube-api-access-fmn7r\" (UniqueName: \"kubernetes.io/projected/c29e722c-2f9e-4f04-acc4-86cc0d364373-kube-api-access-fmn7r\") pod \"c29e722c-2f9e-4f04-acc4-86cc0d364373\" (UID: \"c29e722c-2f9e-4f04-acc4-86cc0d364373\") " Oct 30 20:29:49.822170 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:49.822150 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c29e722c-2f9e-4f04-acc4-86cc0d364373-tls-certs\" (UniqueName: \"kubernetes.io/secret/c29e722c-2f9e-4f04-acc4-86cc0d364373-tls-certs\") pod \"c29e722c-2f9e-4f04-acc4-86cc0d364373\" (UID: \"c29e722c-2f9e-4f04-acc4-86cc0d364373\") " Oct 30 20:29:49.822335 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:49.822236 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c29e722c-2f9e-4f04-acc4-86cc0d364373-tmp-dir" pod "c29e722c-2f9e-4f04-acc4-86cc0d364373" (UID: "c29e722c-2f9e-4f04-acc4-86cc0d364373"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:29:49.822335 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:49.822313 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c29e722c-2f9e-4f04-acc4-86cc0d364373-model-cache" pod "c29e722c-2f9e-4f04-acc4-86cc0d364373" (UID: "c29e722c-2f9e-4f04-acc4-86cc0d364373"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:29:49.822446 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:49.822374 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c29e722c-2f9e-4f04-acc4-86cc0d364373-home" pod "c29e722c-2f9e-4f04-acc4-86cc0d364373" (UID: "c29e722c-2f9e-4f04-acc4-86cc0d364373"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:29:49.822606 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:49.822581 4292 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c29e722c-2f9e-4f04-acc4-86cc0d364373-tmp-dir\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:29:49.822606 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:49.822598 4292 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c29e722c-2f9e-4f04-acc4-86cc0d364373-model-cache\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:29:49.822606 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:49.822608 4292 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c29e722c-2f9e-4f04-acc4-86cc0d364373-home\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:29:49.824324 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:49.824300 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c29e722c-2f9e-4f04-acc4-86cc0d364373-kube-api-access-fmn7r" pod "c29e722c-2f9e-4f04-acc4-86cc0d364373" (UID: "c29e722c-2f9e-4f04-acc4-86cc0d364373"). InnerVolumeSpecName "kube-api-access-fmn7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:29:49.824455 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:49.824325 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c29e722c-2f9e-4f04-acc4-86cc0d364373-dshm" pod "c29e722c-2f9e-4f04-acc4-86cc0d364373" (UID: "c29e722c-2f9e-4f04-acc4-86cc0d364373"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:29:49.824455 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:49.824338 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c29e722c-2f9e-4f04-acc4-86cc0d364373-tls-certs" pod "c29e722c-2f9e-4f04-acc4-86cc0d364373" (UID: "c29e722c-2f9e-4f04-acc4-86cc0d364373"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:29:49.923766 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:49.923736 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmn7r\" (UniqueName: \"kubernetes.io/projected/c29e722c-2f9e-4f04-acc4-86cc0d364373-kube-api-access-fmn7r\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:29:49.923766 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:49.923763 4292 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c29e722c-2f9e-4f04-acc4-86cc0d364373-tls-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:29:49.923766 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:49.923774 4292 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c29e722c-2f9e-4f04-acc4-86cc0d364373-dshm\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:29:50.535257 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:50.535218 4292 generic.go:356] "Generic (PLEG): container finished" podID="c29e722c-2f9e-4f04-acc4-86cc0d364373" containerID="ff9e5164fd7e5c208041fe711c5e99fa6384e3393ea3ca3ca45afaf4ba19ff7f" exitCode=0 Oct 30 20:29:50.535785 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:50.535269 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-056a27/auth-test-model-056a27-kserve-bdf98c7c-k7tgb" Oct 30 20:29:50.535785 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:50.535298 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-056a27/auth-test-model-056a27-kserve-bdf98c7c-k7tgb" event={"ID":"c29e722c-2f9e-4f04-acc4-86cc0d364373","Type":"ContainerDied","Data":"ff9e5164fd7e5c208041fe711c5e99fa6384e3393ea3ca3ca45afaf4ba19ff7f"} Oct 30 20:29:50.535785 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:50.535343 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-056a27/auth-test-model-056a27-kserve-bdf98c7c-k7tgb" event={"ID":"c29e722c-2f9e-4f04-acc4-86cc0d364373","Type":"ContainerDied","Data":"cdf4c6b79c3beebe98f30d49bd2f50bff86bf084bdfb89b533aa8d26296d8f57"} Oct 30 20:29:50.535785 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:50.535378 4292 scope.go:122] "RemoveContainer" containerID="ff9e5164fd7e5c208041fe711c5e99fa6384e3393ea3ca3ca45afaf4ba19ff7f" Oct 30 20:29:50.545892 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:50.545870 4292 scope.go:122] "RemoveContainer" containerID="ff9e5164fd7e5c208041fe711c5e99fa6384e3393ea3ca3ca45afaf4ba19ff7f" Oct 30 20:29:50.546147 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:29:50.546125 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff9e5164fd7e5c208041fe711c5e99fa6384e3393ea3ca3ca45afaf4ba19ff7f\": container with ID starting with ff9e5164fd7e5c208041fe711c5e99fa6384e3393ea3ca3ca45afaf4ba19ff7f not found: ID does not exist" containerID="ff9e5164fd7e5c208041fe711c5e99fa6384e3393ea3ca3ca45afaf4ba19ff7f" Oct 30 20:29:50.546243 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:50.546153 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff9e5164fd7e5c208041fe711c5e99fa6384e3393ea3ca3ca45afaf4ba19ff7f"} err="failed to get container status \"ff9e5164fd7e5c208041fe711c5e99fa6384e3393ea3ca3ca45afaf4ba19ff7f\": rpc error: code = NotFound desc = could not find container \"ff9e5164fd7e5c208041fe711c5e99fa6384e3393ea3ca3ca45afaf4ba19ff7f\": container with ID starting with ff9e5164fd7e5c208041fe711c5e99fa6384e3393ea3ca3ca45afaf4ba19ff7f not found: ID does not exist" Oct 30 20:29:50.654187 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:50.654139 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-056a27/auth-test-model-056a27-kserve-bdf98c7c-k7tgb" podUID="c29e722c-2f9e-4f04-acc4-86cc0d364373" containerName="main" probeResult="failure" output="Get \"https://10.133.0.160:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:29:50.705184 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:50.705149 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-056a27/auth-test-model-056a27-kserve-bdf98c7c-k7tgb"] Oct 30 20:29:50.755837 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:50.755805 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-056a27/auth-test-model-056a27-kserve-bdf98c7c-k7tgb"] Oct 30 20:29:52.581835 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:52.581805 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c29e722c-2f9e-4f04-acc4-86cc0d364373" path="/var/lib/kubelet/pods/c29e722c-2f9e-4f04-acc4-86cc0d364373/volumes" Oct 30 20:29:57.928431 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:29:57.928399 4292 scope.go:122] "RemoveContainer" containerID="a45731d1a42e3bc053db21d22a577915714f643f308769089648dd39c9e710cc" Oct 30 20:30:00.182759 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.182727 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64120b-29889870-jps8m"] Oct 30 20:30:00.183305 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.183286 4292 cpu_manager.go:414] "containerMap: removing container" podUID="c29e722c-2f9e-4f04-acc4-86cc0d364373" containerName="main" Oct 30 20:30:00.183402 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.183307 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c29e722c-2f9e-4f04-acc4-86cc0d364373" containerName="main" Oct 30 20:30:00.183460 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.183450 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="c29e722c-2f9e-4f04-acc4-86cc0d364373" containerName="main" Oct 30 20:30:00.188524 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.188504 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-b37684-29889870-x8xw2"] Oct 30 20:30:00.188641 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.188614 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64120b-29889870-jps8m" Oct 30 20:30:00.192087 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.192071 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-b37684-29889870-x8xw2" Oct 30 20:30:00.198267 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.197977 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-056a27-29889870-4tb2q"] Oct 30 20:30:00.201845 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.201829 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-056a27-29889870-4tb2q" Oct 30 20:30:00.246381 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.243012 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79ffe45a-2987-49cc-909c-920ccbbbb53a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-056a27-29889870-4tb2q\" (UID: \"79ffe45a-2987-49cc-909c-920ccbbbb53a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-056a27-29889870-4tb2q" Oct 30 20:30:00.246381 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.243076 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nkff\" (UniqueName: \"kubernetes.io/projected/a0b2551f-95d5-4a8a-b6e5-08cd9a4d9572-kube-api-access-7nkff\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-b37684-29889870-x8xw2\" (UID: \"a0b2551f-95d5-4a8a-b6e5-08cd9a4d9572\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-b37684-29889870-x8xw2" Oct 30 20:30:00.246381 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.243120 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b2551f-95d5-4a8a-b6e5-08cd9a4d9572-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-b37684-29889870-x8xw2\" (UID: \"a0b2551f-95d5-4a8a-b6e5-08cd9a4d9572\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-b37684-29889870-x8xw2" Oct 30 20:30:00.246381 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.243267 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z9wh\" (UniqueName: \"kubernetes.io/projected/9d3d5c8c-e915-424c-a674-07c2bfb01a7d-kube-api-access-6z9wh\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-64120b-29889870-jps8m\" (UID: \"9d3d5c8c-e915-424c-a674-07c2bfb01a7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64120b-29889870-jps8m" Oct 30 20:30:00.246381 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.243297 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phmbw\" (UniqueName: \"kubernetes.io/projected/79ffe45a-2987-49cc-909c-920ccbbbb53a-kube-api-access-phmbw\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-056a27-29889870-4tb2q\" (UID: \"79ffe45a-2987-49cc-909c-920ccbbbb53a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-056a27-29889870-4tb2q" Oct 30 20:30:00.246381 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.243366 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d3d5c8c-e915-424c-a674-07c2bfb01a7d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-64120b-29889870-jps8m\" (UID: \"9d3d5c8c-e915-424c-a674-07c2bfb01a7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64120b-29889870-jps8m" Oct 30 20:30:00.246381 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.243924 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e4864d-29889870-djv4g"] Oct 30 20:30:00.248964 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.248940 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e4864d-29889870-djv4g" Oct 30 20:30:00.275377 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.269453 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0e72bd-29889870-444qs"] Oct 30 20:30:00.296458 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.296423 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\"" Oct 30 20:30:00.340763 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.338548 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889870-f6rv5"] Oct 30 20:30:00.340763 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.340188 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0e72bd-29889870-444qs" Oct 30 20:30:00.344910 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.344811 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phmbw\" (UniqueName: \"kubernetes.io/projected/79ffe45a-2987-49cc-909c-920ccbbbb53a-kube-api-access-phmbw\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-056a27-29889870-4tb2q\" (UID: \"79ffe45a-2987-49cc-909c-920ccbbbb53a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-056a27-29889870-4tb2q" Oct 30 20:30:00.344910 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.344886 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d3d5c8c-e915-424c-a674-07c2bfb01a7d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-64120b-29889870-jps8m\" (UID: \"9d3d5c8c-e915-424c-a674-07c2bfb01a7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64120b-29889870-jps8m" Oct 30 20:30:00.345117 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.344949 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpgkg\" (UniqueName: \"kubernetes.io/projected/a895dc65-411f-468e-8872-3fb6b6e35a5e-kube-api-access-tpgkg\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-e4864d-29889870-djv4g\" (UID: \"a895dc65-411f-468e-8872-3fb6b6e35a5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e4864d-29889870-djv4g" Oct 30 20:30:00.345117 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.344986 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79ffe45a-2987-49cc-909c-920ccbbbb53a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-056a27-29889870-4tb2q\" (UID: \"79ffe45a-2987-49cc-909c-920ccbbbb53a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-056a27-29889870-4tb2q" Oct 30 20:30:00.347805 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.345817 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d3d5c8c-e915-424c-a674-07c2bfb01a7d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-64120b-29889870-jps8m\" (UID: \"9d3d5c8c-e915-424c-a674-07c2bfb01a7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64120b-29889870-jps8m" Oct 30 20:30:00.347805 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.346969 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79ffe45a-2987-49cc-909c-920ccbbbb53a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-056a27-29889870-4tb2q\" (UID: \"79ffe45a-2987-49cc-909c-920ccbbbb53a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-056a27-29889870-4tb2q" Oct 30 20:30:00.347805 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.347107 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cafbb2-29889870-q7kd5"] Oct 30 20:30:00.347805 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.347253 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-f6rv5" Oct 30 20:30:00.348330 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.348311 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nkff\" (UniqueName: \"kubernetes.io/projected/a0b2551f-95d5-4a8a-b6e5-08cd9a4d9572-kube-api-access-7nkff\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-b37684-29889870-x8xw2\" (UID: \"a0b2551f-95d5-4a8a-b6e5-08cd9a4d9572\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-b37684-29889870-x8xw2" Oct 30 20:30:00.348793 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.348710 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b2551f-95d5-4a8a-b6e5-08cd9a4d9572-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-b37684-29889870-x8xw2\" (UID: \"a0b2551f-95d5-4a8a-b6e5-08cd9a4d9572\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-b37684-29889870-x8xw2" Oct 30 20:30:00.361679 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.361652 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a895dc65-411f-468e-8872-3fb6b6e35a5e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-e4864d-29889870-djv4g\" (UID: \"a895dc65-411f-468e-8872-3fb6b6e35a5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e4864d-29889870-djv4g" Oct 30 20:30:00.380929 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.380897 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z9wh\" (UniqueName: \"kubernetes.io/projected/9d3d5c8c-e915-424c-a674-07c2bfb01a7d-kube-api-access-6z9wh\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-64120b-29889870-jps8m\" (UID: \"9d3d5c8c-e915-424c-a674-07c2bfb01a7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64120b-29889870-jps8m" Oct 30 20:30:00.381189 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.366932 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b2551f-95d5-4a8a-b6e5-08cd9a4d9572-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-b37684-29889870-x8xw2\" (UID: \"a0b2551f-95d5-4a8a-b6e5-08cd9a4d9572\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-b37684-29889870-x8xw2" Oct 30 20:30:00.398373 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.387689 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889870-zws7m"] Oct 30 20:30:00.398373 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.395467 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cafbb2-29889870-q7kd5" Oct 30 20:30:00.400192 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.400166 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-487c5f-29889870-pljq2"] Oct 30 20:30:00.401087 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.401060 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889870-zws7m" Oct 30 20:30:00.405664 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.405642 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-056a27-29889870-4tb2q"] Oct 30 20:30:00.405815 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.405691 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64120b-29889870-jps8m"] Oct 30 20:30:00.405815 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.405718 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-b37684-29889870-x8xw2"] Oct 30 20:30:00.405815 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.405750 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e4864d-29889870-djv4g"] Oct 30 20:30:00.405815 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.405773 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889870-f6rv5"] Oct 30 20:30:00.405815 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.405795 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-487c5f-29889870-pljq2"] Oct 30 20:30:00.405815 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.405817 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889870-zws7m"] Oct 30 20:30:00.406135 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.405900 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-487c5f-29889870-pljq2" Oct 30 20:30:00.414418 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.411990 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phmbw\" (UniqueName: \"kubernetes.io/projected/79ffe45a-2987-49cc-909c-920ccbbbb53a-kube-api-access-phmbw\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-056a27-29889870-4tb2q\" (UID: \"79ffe45a-2987-49cc-909c-920ccbbbb53a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-056a27-29889870-4tb2q" Oct 30 20:30:00.432339 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.429871 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cafbb2-29889870-q7kd5"] Oct 30 20:30:00.432339 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.432270 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nkff\" (UniqueName: \"kubernetes.io/projected/a0b2551f-95d5-4a8a-b6e5-08cd9a4d9572-kube-api-access-7nkff\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-b37684-29889870-x8xw2\" (UID: \"a0b2551f-95d5-4a8a-b6e5-08cd9a4d9572\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-b37684-29889870-x8xw2" Oct 30 20:30:00.434033 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.433974 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0e72bd-29889870-444qs"] Oct 30 20:30:00.449002 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.448539 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z9wh\" (UniqueName: \"kubernetes.io/projected/9d3d5c8c-e915-424c-a674-07c2bfb01a7d-kube-api-access-6z9wh\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-64120b-29889870-jps8m\" (UID: \"9d3d5c8c-e915-424c-a674-07c2bfb01a7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64120b-29889870-jps8m" Oct 30 20:30:00.485076 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.485031 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff6b576c-bc3c-4a15-ae30-57474f6285d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-0e72bd-29889870-444qs\" (UID: \"ff6b576c-bc3c-4a15-ae30-57474f6285d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0e72bd-29889870-444qs" Oct 30 20:30:00.485242 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.485111 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6bqv\" (UniqueName: \"kubernetes.io/projected/57e5a5eb-f9ae-42f6-8c6e-e4336b81c953-kube-api-access-b6bqv\") pod \"maas-api-key-cleanup-e2e-worker-w2-b0a183-29889870-zws7m\" (UID: \"57e5a5eb-f9ae-42f6-8c6e-e4336b81c953\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889870-zws7m" Oct 30 20:30:00.485332 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.485245 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8rgk\" (UniqueName: \"kubernetes.io/projected/062d2841-086a-493c-8bd4-e26a0f3c82ff-kube-api-access-s8rgk\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-cafbb2-29889870-q7kd5\" (UID: \"062d2841-086a-493c-8bd4-e26a0f3c82ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cafbb2-29889870-q7kd5" Oct 30 20:30:00.485332 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.485270 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g872\" (UniqueName: \"kubernetes.io/projected/32b36613-26a9-4472-90fd-916bc95383b6-kube-api-access-6g872\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-487c5f-29889870-pljq2\" (UID: \"32b36613-26a9-4472-90fd-916bc95383b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-487c5f-29889870-pljq2" Oct 30 20:30:00.485332 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.485306 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpgkg\" (UniqueName: \"kubernetes.io/projected/a895dc65-411f-468e-8872-3fb6b6e35a5e-kube-api-access-tpgkg\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-e4864d-29889870-djv4g\" (UID: \"a895dc65-411f-468e-8872-3fb6b6e35a5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e4864d-29889870-djv4g" Oct 30 20:30:00.485576 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.485534 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sctzl\" (UniqueName: \"kubernetes.io/projected/ff6b576c-bc3c-4a15-ae30-57474f6285d0-kube-api-access-sctzl\") pod \"maas-api-key-cleanup-e2e-extmod-a-0e72bd-29889870-444qs\" (UID: \"ff6b576c-bc3c-4a15-ae30-57474f6285d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0e72bd-29889870-444qs" Oct 30 20:30:00.485643 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.485580 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xmpt\" (UniqueName: \"kubernetes.io/projected/bc4d4a92-2ef3-42a5-82be-70bb892011b1-kube-api-access-8xmpt\") pod \"maas-api-key-cleanup-29889870-f6rv5\" (UID: \"bc4d4a92-2ef3-42a5-82be-70bb892011b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-f6rv5" Oct 30 20:30:00.485643 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.485606 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc4d4a92-2ef3-42a5-82be-70bb892011b1-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889870-f6rv5\" (UID: \"bc4d4a92-2ef3-42a5-82be-70bb892011b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-f6rv5" Oct 30 20:30:00.485643 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.485632 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/062d2841-086a-493c-8bd4-e26a0f3c82ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-cafbb2-29889870-q7kd5\" (UID: \"062d2841-086a-493c-8bd4-e26a0f3c82ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cafbb2-29889870-q7kd5" Oct 30 20:30:00.485812 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.485725 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a895dc65-411f-468e-8872-3fb6b6e35a5e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-e4864d-29889870-djv4g\" (UID: \"a895dc65-411f-468e-8872-3fb6b6e35a5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e4864d-29889870-djv4g" Oct 30 20:30:00.485812 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.485780 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32b36613-26a9-4472-90fd-916bc95383b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-487c5f-29889870-pljq2\" (UID: \"32b36613-26a9-4472-90fd-916bc95383b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-487c5f-29889870-pljq2" Oct 30 20:30:00.485928 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.485810 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57e5a5eb-f9ae-42f6-8c6e-e4336b81c953-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-b0a183-29889870-zws7m\" (UID: \"57e5a5eb-f9ae-42f6-8c6e-e4336b81c953\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889870-zws7m" Oct 30 20:30:00.486707 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.486679 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a895dc65-411f-468e-8872-3fb6b6e35a5e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-e4864d-29889870-djv4g\" (UID: \"a895dc65-411f-468e-8872-3fb6b6e35a5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e4864d-29889870-djv4g" Oct 30 20:30:00.508591 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.505242 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64120b-29889870-jps8m" Oct 30 20:30:00.517505 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.517477 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-b37684-29889870-x8xw2" Oct 30 20:30:00.547771 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.547564 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-056a27-29889870-4tb2q" Oct 30 20:30:00.596786 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.591130 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpgkg\" (UniqueName: \"kubernetes.io/projected/a895dc65-411f-468e-8872-3fb6b6e35a5e-kube-api-access-tpgkg\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-e4864d-29889870-djv4g\" (UID: \"a895dc65-411f-468e-8872-3fb6b6e35a5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e4864d-29889870-djv4g" Oct 30 20:30:00.596786 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.593209 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sctzl\" (UniqueName: \"kubernetes.io/projected/ff6b576c-bc3c-4a15-ae30-57474f6285d0-kube-api-access-sctzl\") pod \"maas-api-key-cleanup-e2e-extmod-a-0e72bd-29889870-444qs\" (UID: \"ff6b576c-bc3c-4a15-ae30-57474f6285d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0e72bd-29889870-444qs" Oct 30 20:30:00.596786 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.593242 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xmpt\" (UniqueName: \"kubernetes.io/projected/bc4d4a92-2ef3-42a5-82be-70bb892011b1-kube-api-access-8xmpt\") pod \"maas-api-key-cleanup-29889870-f6rv5\" (UID: \"bc4d4a92-2ef3-42a5-82be-70bb892011b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-f6rv5" Oct 30 20:30:00.596786 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.593258 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc4d4a92-2ef3-42a5-82be-70bb892011b1-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889870-f6rv5\" (UID: \"bc4d4a92-2ef3-42a5-82be-70bb892011b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-f6rv5" Oct 30 20:30:00.596786 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.593275 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/062d2841-086a-493c-8bd4-e26a0f3c82ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-cafbb2-29889870-q7kd5\" (UID: \"062d2841-086a-493c-8bd4-e26a0f3c82ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cafbb2-29889870-q7kd5" Oct 30 20:30:00.596786 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.593298 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32b36613-26a9-4472-90fd-916bc95383b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-487c5f-29889870-pljq2\" (UID: \"32b36613-26a9-4472-90fd-916bc95383b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-487c5f-29889870-pljq2" Oct 30 20:30:00.596786 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.593314 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57e5a5eb-f9ae-42f6-8c6e-e4336b81c953-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-b0a183-29889870-zws7m\" (UID: \"57e5a5eb-f9ae-42f6-8c6e-e4336b81c953\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889870-zws7m" Oct 30 20:30:00.596786 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.593436 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff6b576c-bc3c-4a15-ae30-57474f6285d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-0e72bd-29889870-444qs\" (UID: \"ff6b576c-bc3c-4a15-ae30-57474f6285d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0e72bd-29889870-444qs" Oct 30 20:30:00.596786 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.593485 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6bqv\" (UniqueName: \"kubernetes.io/projected/57e5a5eb-f9ae-42f6-8c6e-e4336b81c953-kube-api-access-b6bqv\") pod \"maas-api-key-cleanup-e2e-worker-w2-b0a183-29889870-zws7m\" (UID: \"57e5a5eb-f9ae-42f6-8c6e-e4336b81c953\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889870-zws7m" Oct 30 20:30:00.596786 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.593554 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8rgk\" (UniqueName: \"kubernetes.io/projected/062d2841-086a-493c-8bd4-e26a0f3c82ff-kube-api-access-s8rgk\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-cafbb2-29889870-q7kd5\" (UID: \"062d2841-086a-493c-8bd4-e26a0f3c82ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cafbb2-29889870-q7kd5" Oct 30 20:30:00.596786 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.593578 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g872\" (UniqueName: \"kubernetes.io/projected/32b36613-26a9-4472-90fd-916bc95383b6-kube-api-access-6g872\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-487c5f-29889870-pljq2\" (UID: \"32b36613-26a9-4472-90fd-916bc95383b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-487c5f-29889870-pljq2" Oct 30 20:30:00.596786 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.594509 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc4d4a92-2ef3-42a5-82be-70bb892011b1-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889870-f6rv5\" (UID: \"bc4d4a92-2ef3-42a5-82be-70bb892011b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-f6rv5" Oct 30 20:30:00.596786 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.594529 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff6b576c-bc3c-4a15-ae30-57474f6285d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-0e72bd-29889870-444qs\" (UID: \"ff6b576c-bc3c-4a15-ae30-57474f6285d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0e72bd-29889870-444qs" Oct 30 20:30:00.596786 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.595206 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32b36613-26a9-4472-90fd-916bc95383b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-487c5f-29889870-pljq2\" (UID: \"32b36613-26a9-4472-90fd-916bc95383b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-487c5f-29889870-pljq2" Oct 30 20:30:00.596786 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.595418 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/062d2841-086a-493c-8bd4-e26a0f3c82ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-cafbb2-29889870-q7kd5\" (UID: \"062d2841-086a-493c-8bd4-e26a0f3c82ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cafbb2-29889870-q7kd5" Oct 30 20:30:00.596786 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.595599 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57e5a5eb-f9ae-42f6-8c6e-e4336b81c953-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-b0a183-29889870-zws7m\" (UID: \"57e5a5eb-f9ae-42f6-8c6e-e4336b81c953\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889870-zws7m" Oct 30 20:30:00.642500 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.641895 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e4864d-29889870-djv4g" Oct 30 20:30:00.687316 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.682510 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8rgk\" (UniqueName: \"kubernetes.io/projected/062d2841-086a-493c-8bd4-e26a0f3c82ff-kube-api-access-s8rgk\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-cafbb2-29889870-q7kd5\" (UID: \"062d2841-086a-493c-8bd4-e26a0f3c82ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cafbb2-29889870-q7kd5" Oct 30 20:30:00.688019 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.687942 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g872\" (UniqueName: \"kubernetes.io/projected/32b36613-26a9-4472-90fd-916bc95383b6-kube-api-access-6g872\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-487c5f-29889870-pljq2\" (UID: \"32b36613-26a9-4472-90fd-916bc95383b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-487c5f-29889870-pljq2" Oct 30 20:30:00.703667 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.702954 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sctzl\" (UniqueName: \"kubernetes.io/projected/ff6b576c-bc3c-4a15-ae30-57474f6285d0-kube-api-access-sctzl\") pod \"maas-api-key-cleanup-e2e-extmod-a-0e72bd-29889870-444qs\" (UID: \"ff6b576c-bc3c-4a15-ae30-57474f6285d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0e72bd-29889870-444qs" Oct 30 20:30:00.703667 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.703617 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6bqv\" (UniqueName: \"kubernetes.io/projected/57e5a5eb-f9ae-42f6-8c6e-e4336b81c953-kube-api-access-b6bqv\") pod \"maas-api-key-cleanup-e2e-worker-w2-b0a183-29889870-zws7m\" (UID: \"57e5a5eb-f9ae-42f6-8c6e-e4336b81c953\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889870-zws7m" Oct 30 20:30:00.704722 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.704519 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xmpt\" (UniqueName: \"kubernetes.io/projected/bc4d4a92-2ef3-42a5-82be-70bb892011b1-kube-api-access-8xmpt\") pod \"maas-api-key-cleanup-29889870-f6rv5\" (UID: \"bc4d4a92-2ef3-42a5-82be-70bb892011b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-f6rv5" Oct 30 20:30:00.705164 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.704970 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-f6rv5" Oct 30 20:30:00.785289 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.757561 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cafbb2-29889870-q7kd5" Oct 30 20:30:00.797517 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.797488 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889870-zws7m" Oct 30 20:30:00.812743 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.812148 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-487c5f-29889870-pljq2" Oct 30 20:30:00.934501 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.934461 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64120b-29889870-jps8m"] Oct 30 20:30:00.984873 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.984842 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-b37684-29889870-x8xw2"] Oct 30 20:30:00.998383 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:00.998335 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0e72bd-29889870-444qs" Oct 30 20:30:01.000238 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:30:00.999372 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0b2551f_95d5_4a8a_b6e5_08cd9a4d9572.slice/crio-396028c7442f124256b5378a20921db1856ec3131f629162eec59c7c17d2869a WatchSource:0}: Error finding container 396028c7442f124256b5378a20921db1856ec3131f629162eec59c7c17d2869a: Status 404 returned error can't find the container with id 396028c7442f124256b5378a20921db1856ec3131f629162eec59c7c17d2869a Oct 30 20:30:01.033629 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:01.033206 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-056a27-29889870-4tb2q"] Oct 30 20:30:01.095008 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:01.094979 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e4864d-29889870-djv4g"] Oct 30 20:30:01.138041 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:01.138007 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889870-f6rv5"] Oct 30 20:30:01.229883 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:30:01.229850 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod062d2841_086a_493c_8bd4_e26a0f3c82ff.slice/crio-7a842cd168c192deff7811a15834099408ab085421a9fe990e6160677ae85ffa WatchSource:0}: Error finding container 7a842cd168c192deff7811a15834099408ab085421a9fe990e6160677ae85ffa: Status 404 returned error can't find the container with id 7a842cd168c192deff7811a15834099408ab085421a9fe990e6160677ae85ffa Oct 30 20:30:01.232566 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:01.232539 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cafbb2-29889870-q7kd5"] Oct 30 20:30:01.246582 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:01.246557 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889870-zws7m"] Oct 30 20:30:01.251446 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:01.251310 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-487c5f-29889870-pljq2"] Oct 30 20:30:01.429111 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:01.429085 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0e72bd-29889870-444qs"] Oct 30 20:30:01.609472 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:01.609444 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-f6rv5" event={"ID":"bc4d4a92-2ef3-42a5-82be-70bb892011b1","Type":"ContainerStarted","Data":"010cc63aedb0d3aad1df9295e084b2f3cb9536c8dd96a304ad3d9b0a882c62f0"} Oct 30 20:30:01.610873 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:01.610843 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e4864d-29889870-djv4g" event={"ID":"a895dc65-411f-468e-8872-3fb6b6e35a5e","Type":"ContainerStarted","Data":"8ccc48c04e4a0f1f04747d5e0da0545d19d2a30df12dccc1f17d196407c37dc5"} Oct 30 20:30:01.612074 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:01.612047 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0e72bd-29889870-444qs" event={"ID":"ff6b576c-bc3c-4a15-ae30-57474f6285d0","Type":"ContainerStarted","Data":"d75a17a892e7c5917fb693b6f51b28e40b85fe24b33225e5b76c2f445406e66f"} Oct 30 20:30:01.613245 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:01.613220 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-056a27-29889870-4tb2q" event={"ID":"79ffe45a-2987-49cc-909c-920ccbbbb53a","Type":"ContainerStarted","Data":"32a0697b187c95a1256569cb1885b45a16a1dd4ce512463c38d39af62d58fc3c"} Oct 30 20:30:01.614510 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:01.614485 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-487c5f-29889870-pljq2" event={"ID":"32b36613-26a9-4472-90fd-916bc95383b6","Type":"ContainerStarted","Data":"52b95354e163a857ea85994f7135795ae256ef3e73ce7835b425cb3d7349da1e"} Oct 30 20:30:01.615622 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:01.615593 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-b37684-29889870-x8xw2" event={"ID":"a0b2551f-95d5-4a8a-b6e5-08cd9a4d9572","Type":"ContainerStarted","Data":"396028c7442f124256b5378a20921db1856ec3131f629162eec59c7c17d2869a"} Oct 30 20:30:01.616818 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:01.616798 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889870-zws7m" event={"ID":"57e5a5eb-f9ae-42f6-8c6e-e4336b81c953","Type":"ContainerStarted","Data":"cdb3213417ce95e5a439cc59d48449d9cfafd2d1b45cf2a9104b845107edd41c"} Oct 30 20:30:01.617944 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:01.617925 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cafbb2-29889870-q7kd5" event={"ID":"062d2841-086a-493c-8bd4-e26a0f3c82ff","Type":"ContainerStarted","Data":"7a842cd168c192deff7811a15834099408ab085421a9fe990e6160677ae85ffa"} Oct 30 20:30:01.619022 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:01.619003 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64120b-29889870-jps8m" event={"ID":"9d3d5c8c-e915-424c-a674-07c2bfb01a7d","Type":"ContainerStarted","Data":"83a3d6be4a6e4608ac251829885f16395b69f74258dd181c6b0bd94da37af46b"} Oct 30 20:30:02.627103 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:02.627068 4292 generic.go:356] "Generic (PLEG): container finished" podID="a0b2551f-95d5-4a8a-b6e5-08cd9a4d9572" containerID="b880c7c0a337af11d22391232e560090bfb3aaeb94d83c9442ec0551b2cef5ea" exitCode=60 Oct 30 20:30:02.627558 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:02.627118 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-b37684-29889870-x8xw2" event={"ID":"a0b2551f-95d5-4a8a-b6e5-08cd9a4d9572","Type":"ContainerDied","Data":"b880c7c0a337af11d22391232e560090bfb3aaeb94d83c9442ec0551b2cef5ea"} Oct 30 20:30:02.627558 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:02.627464 4292 scope.go:122] "RemoveContainer" containerID="b880c7c0a337af11d22391232e560090bfb3aaeb94d83c9442ec0551b2cef5ea" Oct 30 20:30:02.628981 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:02.628957 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889870-zws7m" event={"ID":"57e5a5eb-f9ae-42f6-8c6e-e4336b81c953","Type":"ContainerStarted","Data":"a06babce7bdc0f0cfd88e2e1946bbbedd7fd4dc161a87d46cd7da99c12c5dd73"} Oct 30 20:30:02.630719 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:02.630631 4292 generic.go:356] "Generic (PLEG): container finished" podID="062d2841-086a-493c-8bd4-e26a0f3c82ff" containerID="2c898098e6b0ffb8ba79c7901ab95beeae27afdf3b818e174d9ae40e69670fc3" exitCode=60 Oct 30 20:30:02.630719 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:02.630675 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cafbb2-29889870-q7kd5" event={"ID":"062d2841-086a-493c-8bd4-e26a0f3c82ff","Type":"ContainerDied","Data":"2c898098e6b0ffb8ba79c7901ab95beeae27afdf3b818e174d9ae40e69670fc3"} Oct 30 20:30:02.630965 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:02.630949 4292 scope.go:122] "RemoveContainer" containerID="2c898098e6b0ffb8ba79c7901ab95beeae27afdf3b818e174d9ae40e69670fc3" Oct 30 20:30:02.632228 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:02.632210 4292 generic.go:356] "Generic (PLEG): container finished" podID="9d3d5c8c-e915-424c-a674-07c2bfb01a7d" containerID="d41e06af92339f9e621cd0ccca9ca5d2ed460b9344a28564268b7648b8809b2b" exitCode=60 Oct 30 20:30:02.632315 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:02.632289 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64120b-29889870-jps8m" event={"ID":"9d3d5c8c-e915-424c-a674-07c2bfb01a7d","Type":"ContainerDied","Data":"d41e06af92339f9e621cd0ccca9ca5d2ed460b9344a28564268b7648b8809b2b"} Oct 30 20:30:02.632501 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:02.632488 4292 scope.go:122] "RemoveContainer" containerID="d41e06af92339f9e621cd0ccca9ca5d2ed460b9344a28564268b7648b8809b2b" Oct 30 20:30:02.633878 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:02.633858 4292 generic.go:356] "Generic (PLEG): container finished" podID="bc4d4a92-2ef3-42a5-82be-70bb892011b1" containerID="5e9b7cd3da4dd079e6555bd9393f88cdd8562476f7e2c2059460ad74aff7afb7" exitCode=60 Oct 30 20:30:02.633963 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:02.633923 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-f6rv5" event={"ID":"bc4d4a92-2ef3-42a5-82be-70bb892011b1","Type":"ContainerDied","Data":"5e9b7cd3da4dd079e6555bd9393f88cdd8562476f7e2c2059460ad74aff7afb7"} Oct 30 20:30:02.634141 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:02.634123 4292 scope.go:122] "RemoveContainer" containerID="5e9b7cd3da4dd079e6555bd9393f88cdd8562476f7e2c2059460ad74aff7afb7" Oct 30 20:30:02.635539 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:02.635521 4292 generic.go:356] "Generic (PLEG): container finished" podID="a895dc65-411f-468e-8872-3fb6b6e35a5e" containerID="e48b28005fa1d4f9bbb639de83d17ca34295061b3d657d97da4a7e39b3924364" exitCode=60 Oct 30 20:30:02.635632 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:02.635605 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e4864d-29889870-djv4g" event={"ID":"a895dc65-411f-468e-8872-3fb6b6e35a5e","Type":"ContainerDied","Data":"e48b28005fa1d4f9bbb639de83d17ca34295061b3d657d97da4a7e39b3924364"} Oct 30 20:30:02.635805 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:02.635790 4292 scope.go:122] "RemoveContainer" containerID="e48b28005fa1d4f9bbb639de83d17ca34295061b3d657d97da4a7e39b3924364" Oct 30 20:30:02.637379 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:02.637334 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0e72bd-29889870-444qs" event={"ID":"ff6b576c-bc3c-4a15-ae30-57474f6285d0","Type":"ContainerStarted","Data":"ed5421e795caecbeba72a341e484dbec3c7e687b3530fe5196184199569cf336"} Oct 30 20:30:02.639750 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:02.639732 4292 generic.go:356] "Generic (PLEG): container finished" podID="79ffe45a-2987-49cc-909c-920ccbbbb53a" containerID="55742d2e2ceb5c545066a3b934d7c0afc6800f8fe97d9fa277f72c498302ac08" exitCode=60 Oct 30 20:30:02.639834 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:02.639811 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-056a27-29889870-4tb2q" event={"ID":"79ffe45a-2987-49cc-909c-920ccbbbb53a","Type":"ContainerDied","Data":"55742d2e2ceb5c545066a3b934d7c0afc6800f8fe97d9fa277f72c498302ac08"} Oct 30 20:30:02.640557 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:02.640181 4292 scope.go:122] "RemoveContainer" containerID="55742d2e2ceb5c545066a3b934d7c0afc6800f8fe97d9fa277f72c498302ac08" Oct 30 20:30:02.641675 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:02.641656 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-487c5f-29889870-pljq2" event={"ID":"32b36613-26a9-4472-90fd-916bc95383b6","Type":"ContainerStarted","Data":"27e8d2f5aa0d5c44820d38c7327788cd82dd3195deea326508ebd371947ada68"} Oct 30 20:30:03.219264 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:03.219179 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0e72bd-29889870-444qs" podStartSLOduration=3.21916473 podStartE2EDuration="3.21916473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:30:03.218068281 +0000 UTC m=+3906.736383739" watchObservedRunningTime="2026-10-30 20:30:03.21916473 +0000 UTC m=+3906.737480187" Oct 30 20:30:03.465375 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:03.462986 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889870-zws7m" podStartSLOduration=3.462965524 podStartE2EDuration="3.462965524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:30:03.453721508 +0000 UTC m=+3906.972036983" watchObservedRunningTime="2026-10-30 20:30:03.462965524 +0000 UTC m=+3906.981280981" Oct 30 20:30:03.687491 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:03.687447 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-b37684-29889870-x8xw2" event={"ID":"a0b2551f-95d5-4a8a-b6e5-08cd9a4d9572","Type":"ContainerStarted","Data":"73946197a6f38ea20b666d2931480aa1af31b7d075f24ef272688de85b0686c4"} Oct 30 20:30:03.690272 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:03.690034 4292 generic.go:356] "Generic (PLEG): container finished" podID="57e5a5eb-f9ae-42f6-8c6e-e4336b81c953" containerID="a06babce7bdc0f0cfd88e2e1946bbbedd7fd4dc161a87d46cd7da99c12c5dd73" exitCode=60 Oct 30 20:30:03.690272 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:03.690133 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889870-zws7m" event={"ID":"57e5a5eb-f9ae-42f6-8c6e-e4336b81c953","Type":"ContainerDied","Data":"a06babce7bdc0f0cfd88e2e1946bbbedd7fd4dc161a87d46cd7da99c12c5dd73"} Oct 30 20:30:03.690857 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:03.690838 4292 scope.go:122] "RemoveContainer" containerID="a06babce7bdc0f0cfd88e2e1946bbbedd7fd4dc161a87d46cd7da99c12c5dd73" Oct 30 20:30:03.703702 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:03.703663 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-487c5f-29889870-pljq2" podStartSLOduration=3.703650566 podStartE2EDuration="3.703650566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:30:03.700246469 +0000 UTC m=+3907.218561928" watchObservedRunningTime="2026-10-30 20:30:03.703650566 +0000 UTC m=+3907.221966023" Oct 30 20:30:03.705961 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:03.705934 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cafbb2-29889870-q7kd5" event={"ID":"062d2841-086a-493c-8bd4-e26a0f3c82ff","Type":"ContainerStarted","Data":"06fdf3e6346498c43375310aba7f1d9dbf748b12c72b53c6866be414c580c7d3"} Oct 30 20:30:03.708092 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:03.708067 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64120b-29889870-jps8m" event={"ID":"9d3d5c8c-e915-424c-a674-07c2bfb01a7d","Type":"ContainerStarted","Data":"accdef60da516f78e5fcb2005e46c35f2b4ea9a5a332764c53868a940cf5a634"} Oct 30 20:30:03.710406 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:03.710385 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-f6rv5" event={"ID":"bc4d4a92-2ef3-42a5-82be-70bb892011b1","Type":"ContainerStarted","Data":"85bbb20cdf12173def845e0ef724a039811f740001b5734ffb28430055cf643d"} Oct 30 20:30:03.712398 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:03.712376 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e4864d-29889870-djv4g" event={"ID":"a895dc65-411f-468e-8872-3fb6b6e35a5e","Type":"ContainerStarted","Data":"0cc7122047009ca0116eed8552ca372be3e1e8985f5c81bcd03a2fe51880aa19"} Oct 30 20:30:03.716399 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:03.714288 4292 generic.go:356] "Generic (PLEG): container finished" podID="ff6b576c-bc3c-4a15-ae30-57474f6285d0" containerID="ed5421e795caecbeba72a341e484dbec3c7e687b3530fe5196184199569cf336" exitCode=60 Oct 30 20:30:03.716399 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:03.714391 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0e72bd-29889870-444qs" event={"ID":"ff6b576c-bc3c-4a15-ae30-57474f6285d0","Type":"ContainerDied","Data":"ed5421e795caecbeba72a341e484dbec3c7e687b3530fe5196184199569cf336"} Oct 30 20:30:03.716801 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:03.716787 4292 scope.go:122] "RemoveContainer" containerID="ed5421e795caecbeba72a341e484dbec3c7e687b3530fe5196184199569cf336" Oct 30 20:30:03.716988 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:03.716968 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-056a27-29889870-4tb2q" event={"ID":"79ffe45a-2987-49cc-909c-920ccbbbb53a","Type":"ContainerStarted","Data":"1db0ade25f1374b29acef8da2c2efc5f28821962b58ac7446b52a09ce7d965bd"} Oct 30 20:30:03.718766 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:03.718745 4292 generic.go:356] "Generic (PLEG): container finished" podID="32b36613-26a9-4472-90fd-916bc95383b6" containerID="27e8d2f5aa0d5c44820d38c7327788cd82dd3195deea326508ebd371947ada68" exitCode=60 Oct 30 20:30:03.718893 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:03.718798 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-487c5f-29889870-pljq2" event={"ID":"32b36613-26a9-4472-90fd-916bc95383b6","Type":"ContainerDied","Data":"27e8d2f5aa0d5c44820d38c7327788cd82dd3195deea326508ebd371947ada68"} Oct 30 20:30:03.719015 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:03.719000 4292 scope.go:122] "RemoveContainer" containerID="27e8d2f5aa0d5c44820d38c7327788cd82dd3195deea326508ebd371947ada68" Oct 30 20:30:03.827426 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:03.827378 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-b37684-29889870-x8xw2" podStartSLOduration=3.827363759 podStartE2EDuration="3.827363759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:30:03.825077457 +0000 UTC m=+3907.343392914" watchObservedRunningTime="2026-10-30 20:30:03.827363759 +0000 UTC m=+3907.345679209" Oct 30 20:30:04.194191 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:04.194138 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e4864d-29889870-djv4g" podStartSLOduration=4.194122505 podStartE2EDuration="4.194122505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:30:04.07497577 +0000 UTC m=+3907.593291230" watchObservedRunningTime="2026-10-30 20:30:04.194122505 +0000 UTC m=+3907.712438012" Oct 30 20:30:04.310050 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:04.310006 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-f6rv5" podStartSLOduration=4.3099926459999995 podStartE2EDuration="4.309992646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:30:04.308928837 +0000 UTC m=+3907.827244314" watchObservedRunningTime="2026-10-30 20:30:04.309992646 +0000 UTC m=+3907.828308148" Oct 30 20:30:04.435779 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:04.435677 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-056a27-29889870-4tb2q" podStartSLOduration=4.435664898 podStartE2EDuration="4.435664898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:30:04.43331648 +0000 UTC m=+3907.951631967" watchObservedRunningTime="2026-10-30 20:30:04.435664898 +0000 UTC m=+3907.953980355" Oct 30 20:30:04.684082 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:04.684030 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64120b-29889870-jps8m" podStartSLOduration=4.684016021 podStartE2EDuration="4.684016021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:30:04.566581867 +0000 UTC m=+3908.084897325" watchObservedRunningTime="2026-10-30 20:30:04.684016021 +0000 UTC m=+3908.202331519" Oct 30 20:30:04.727501 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:04.727404 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0e72bd-29889870-444qs" event={"ID":"ff6b576c-bc3c-4a15-ae30-57474f6285d0","Type":"ContainerStarted","Data":"ef42c81409c6ad94344985ba6c22f62035870189adb72e70fca7afe21d9176ee"} Oct 30 20:30:04.729197 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:04.729169 4292 generic.go:356] "Generic (PLEG): container finished" podID="79ffe45a-2987-49cc-909c-920ccbbbb53a" containerID="1db0ade25f1374b29acef8da2c2efc5f28821962b58ac7446b52a09ce7d965bd" exitCode=60 Oct 30 20:30:04.729377 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:04.729233 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-056a27-29889870-4tb2q" event={"ID":"79ffe45a-2987-49cc-909c-920ccbbbb53a","Type":"ContainerDied","Data":"1db0ade25f1374b29acef8da2c2efc5f28821962b58ac7446b52a09ce7d965bd"} Oct 30 20:30:04.729377 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:04.729287 4292 scope.go:122] "RemoveContainer" containerID="55742d2e2ceb5c545066a3b934d7c0afc6800f8fe97d9fa277f72c498302ac08" Oct 30 20:30:04.729563 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:04.729545 4292 scope.go:122] "RemoveContainer" containerID="1db0ade25f1374b29acef8da2c2efc5f28821962b58ac7446b52a09ce7d965bd" Oct 30 20:30:04.729874 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:30:04.729837 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-056a27-29889870-4tb2q_odh-ai-gateway-infra(79ffe45a-2987-49cc-909c-920ccbbbb53a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-056a27-29889870-4tb2q" podUID="79ffe45a-2987-49cc-909c-920ccbbbb53a" Oct 30 20:30:04.731321 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:04.731299 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-487c5f-29889870-pljq2" event={"ID":"32b36613-26a9-4472-90fd-916bc95383b6","Type":"ContainerStarted","Data":"f36bb8a65af8ad97b7be284fd2b47f5b673abdb6eb2aa6435edc397aa9ba8109"} Oct 30 20:30:04.733087 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:04.733068 4292 generic.go:356] "Generic (PLEG): container finished" podID="a0b2551f-95d5-4a8a-b6e5-08cd9a4d9572" containerID="73946197a6f38ea20b666d2931480aa1af31b7d075f24ef272688de85b0686c4" exitCode=60 Oct 30 20:30:04.733203 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:04.733146 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-b37684-29889870-x8xw2" event={"ID":"a0b2551f-95d5-4a8a-b6e5-08cd9a4d9572","Type":"ContainerDied","Data":"73946197a6f38ea20b666d2931480aa1af31b7d075f24ef272688de85b0686c4"} Oct 30 20:30:04.733318 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:04.733303 4292 scope.go:122] "RemoveContainer" containerID="73946197a6f38ea20b666d2931480aa1af31b7d075f24ef272688de85b0686c4" Oct 30 20:30:04.733575 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:30:04.733549 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-b37684-29889870-x8xw2_odh-ai-gateway-infra(a0b2551f-95d5-4a8a-b6e5-08cd9a4d9572)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-b37684-29889870-x8xw2" podUID="a0b2551f-95d5-4a8a-b6e5-08cd9a4d9572" Oct 30 20:30:04.734869 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:04.734848 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889870-zws7m" event={"ID":"57e5a5eb-f9ae-42f6-8c6e-e4336b81c953","Type":"ContainerStarted","Data":"317fa17ca05ebb4a72203415fc083167430b21d87f237a48963a35e91e12c369"} Oct 30 20:30:04.736718 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:04.736693 4292 generic.go:356] "Generic (PLEG): container finished" podID="062d2841-086a-493c-8bd4-e26a0f3c82ff" containerID="06fdf3e6346498c43375310aba7f1d9dbf748b12c72b53c6866be414c580c7d3" exitCode=60 Oct 30 20:30:04.736805 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:04.736770 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cafbb2-29889870-q7kd5" event={"ID":"062d2841-086a-493c-8bd4-e26a0f3c82ff","Type":"ContainerDied","Data":"06fdf3e6346498c43375310aba7f1d9dbf748b12c72b53c6866be414c580c7d3"} Oct 30 20:30:04.736952 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:04.736937 4292 scope.go:122] "RemoveContainer" containerID="06fdf3e6346498c43375310aba7f1d9dbf748b12c72b53c6866be414c580c7d3" Oct 30 20:30:04.737177 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:30:04.737153 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-cafbb2-29889870-q7kd5_odh-ai-gateway-infra(062d2841-086a-493c-8bd4-e26a0f3c82ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cafbb2-29889870-q7kd5" podUID="062d2841-086a-493c-8bd4-e26a0f3c82ff" Oct 30 20:30:04.738694 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:04.738670 4292 generic.go:356] "Generic (PLEG): container finished" podID="9d3d5c8c-e915-424c-a674-07c2bfb01a7d" containerID="accdef60da516f78e5fcb2005e46c35f2b4ea9a5a332764c53868a940cf5a634" exitCode=60 Oct 30 20:30:04.738781 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:04.738732 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64120b-29889870-jps8m" event={"ID":"9d3d5c8c-e915-424c-a674-07c2bfb01a7d","Type":"ContainerDied","Data":"accdef60da516f78e5fcb2005e46c35f2b4ea9a5a332764c53868a940cf5a634"} Oct 30 20:30:04.739131 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:04.739114 4292 scope.go:122] "RemoveContainer" containerID="accdef60da516f78e5fcb2005e46c35f2b4ea9a5a332764c53868a940cf5a634" Oct 30 20:30:04.739403 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:30:04.739374 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-64120b-29889870-jps8m_odh-ai-gateway-infra(9d3d5c8c-e915-424c-a674-07c2bfb01a7d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64120b-29889870-jps8m" podUID="9d3d5c8c-e915-424c-a674-07c2bfb01a7d" Oct 30 20:30:04.740663 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:04.740643 4292 generic.go:356] "Generic (PLEG): container finished" podID="bc4d4a92-2ef3-42a5-82be-70bb892011b1" containerID="85bbb20cdf12173def845e0ef724a039811f740001b5734ffb28430055cf643d" exitCode=60 Oct 30 20:30:04.740761 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:04.740706 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-f6rv5" event={"ID":"bc4d4a92-2ef3-42a5-82be-70bb892011b1","Type":"ContainerDied","Data":"85bbb20cdf12173def845e0ef724a039811f740001b5734ffb28430055cf643d"} Oct 30 20:30:04.740964 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:04.740946 4292 scope.go:122] "RemoveContainer" containerID="85bbb20cdf12173def845e0ef724a039811f740001b5734ffb28430055cf643d" Oct 30 20:30:04.741161 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:30:04.741134 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889870-f6rv5_odh-ai-gateway-infra(bc4d4a92-2ef3-42a5-82be-70bb892011b1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-f6rv5" podUID="bc4d4a92-2ef3-42a5-82be-70bb892011b1" Oct 30 20:30:04.742614 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:04.742593 4292 generic.go:356] "Generic (PLEG): container finished" podID="a895dc65-411f-468e-8872-3fb6b6e35a5e" containerID="0cc7122047009ca0116eed8552ca372be3e1e8985f5c81bcd03a2fe51880aa19" exitCode=60 Oct 30 20:30:04.742703 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:04.742664 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e4864d-29889870-djv4g" event={"ID":"a895dc65-411f-468e-8872-3fb6b6e35a5e","Type":"ContainerDied","Data":"0cc7122047009ca0116eed8552ca372be3e1e8985f5c81bcd03a2fe51880aa19"} Oct 30 20:30:04.742813 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:04.742800 4292 scope.go:122] "RemoveContainer" containerID="0cc7122047009ca0116eed8552ca372be3e1e8985f5c81bcd03a2fe51880aa19" Oct 30 20:30:04.742979 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:30:04.742960 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-e4864d-29889870-djv4g_odh-ai-gateway-infra(a895dc65-411f-468e-8872-3fb6b6e35a5e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e4864d-29889870-djv4g" podUID="a895dc65-411f-468e-8872-3fb6b6e35a5e" Oct 30 20:30:04.743043 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:04.743027 4292 scope.go:122] "RemoveContainer" containerID="b880c7c0a337af11d22391232e560090bfb3aaeb94d83c9442ec0551b2cef5ea" Oct 30 20:30:04.781197 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:04.781172 4292 scope.go:122] "RemoveContainer" containerID="2c898098e6b0ffb8ba79c7901ab95beeae27afdf3b818e174d9ae40e69670fc3" Oct 30 20:30:04.803297 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:04.803253 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cafbb2-29889870-q7kd5" podStartSLOduration=4.803239856 podStartE2EDuration="4.803239856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:30:04.800386181 +0000 UTC m=+3908.318701660" watchObservedRunningTime="2026-10-30 20:30:04.803239856 +0000 UTC m=+3908.321555312" Oct 30 20:30:04.828452 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:04.828429 4292 scope.go:122] "RemoveContainer" containerID="d41e06af92339f9e621cd0ccca9ca5d2ed460b9344a28564268b7648b8809b2b" Oct 30 20:30:04.841279 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:04.841256 4292 scope.go:122] "RemoveContainer" containerID="5e9b7cd3da4dd079e6555bd9393f88cdd8562476f7e2c2059460ad74aff7afb7" Oct 30 20:30:04.881436 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:04.881406 4292 scope.go:122] "RemoveContainer" containerID="e48b28005fa1d4f9bbb639de83d17ca34295061b3d657d97da4a7e39b3924364" Oct 30 20:30:05.779249 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:05.779216 4292 scope.go:122] "RemoveContainer" containerID="85bbb20cdf12173def845e0ef724a039811f740001b5734ffb28430055cf643d" Oct 30 20:30:05.780159 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:30:05.780129 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889870-f6rv5_odh-ai-gateway-infra(bc4d4a92-2ef3-42a5-82be-70bb892011b1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-f6rv5" podUID="bc4d4a92-2ef3-42a5-82be-70bb892011b1" Oct 30 20:30:05.787146 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:05.787125 4292 scope.go:122] "RemoveContainer" containerID="0cc7122047009ca0116eed8552ca372be3e1e8985f5c81bcd03a2fe51880aa19" Oct 30 20:30:05.787566 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:30:05.787542 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-e4864d-29889870-djv4g_odh-ai-gateway-infra(a895dc65-411f-468e-8872-3fb6b6e35a5e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e4864d-29889870-djv4g" podUID="a895dc65-411f-468e-8872-3fb6b6e35a5e" Oct 30 20:30:05.789478 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:05.789445 4292 generic.go:356] "Generic (PLEG): container finished" podID="ff6b576c-bc3c-4a15-ae30-57474f6285d0" containerID="ef42c81409c6ad94344985ba6c22f62035870189adb72e70fca7afe21d9176ee" exitCode=60 Oct 30 20:30:05.789579 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:05.789527 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0e72bd-29889870-444qs" event={"ID":"ff6b576c-bc3c-4a15-ae30-57474f6285d0","Type":"ContainerDied","Data":"ef42c81409c6ad94344985ba6c22f62035870189adb72e70fca7afe21d9176ee"} Oct 30 20:30:05.789641 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:05.789625 4292 scope.go:122] "RemoveContainer" containerID="ed5421e795caecbeba72a341e484dbec3c7e687b3530fe5196184199569cf336" Oct 30 20:30:05.789747 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:05.789733 4292 scope.go:122] "RemoveContainer" containerID="ef42c81409c6ad94344985ba6c22f62035870189adb72e70fca7afe21d9176ee" Oct 30 20:30:05.789963 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:30:05.789937 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-0e72bd-29889870-444qs_odh-ai-gateway-infra(ff6b576c-bc3c-4a15-ae30-57474f6285d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0e72bd-29889870-444qs" podUID="ff6b576c-bc3c-4a15-ae30-57474f6285d0" Oct 30 20:30:05.791346 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:05.791332 4292 scope.go:122] "RemoveContainer" containerID="1db0ade25f1374b29acef8da2c2efc5f28821962b58ac7446b52a09ce7d965bd" Oct 30 20:30:05.791549 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:30:05.791529 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-056a27-29889870-4tb2q_odh-ai-gateway-infra(79ffe45a-2987-49cc-909c-920ccbbbb53a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-056a27-29889870-4tb2q" podUID="79ffe45a-2987-49cc-909c-920ccbbbb53a" Oct 30 20:30:05.792835 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:05.792815 4292 generic.go:356] "Generic (PLEG): container finished" podID="32b36613-26a9-4472-90fd-916bc95383b6" containerID="f36bb8a65af8ad97b7be284fd2b47f5b673abdb6eb2aa6435edc397aa9ba8109" exitCode=60 Oct 30 20:30:05.792923 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:05.792900 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-487c5f-29889870-pljq2" event={"ID":"32b36613-26a9-4472-90fd-916bc95383b6","Type":"ContainerDied","Data":"f36bb8a65af8ad97b7be284fd2b47f5b673abdb6eb2aa6435edc397aa9ba8109"} Oct 30 20:30:05.793072 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:05.793056 4292 scope.go:122] "RemoveContainer" containerID="f36bb8a65af8ad97b7be284fd2b47f5b673abdb6eb2aa6435edc397aa9ba8109" Oct 30 20:30:05.793256 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:30:05.793236 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-487c5f-29889870-pljq2_odh-ai-gateway-infra(32b36613-26a9-4472-90fd-916bc95383b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-487c5f-29889870-pljq2" podUID="32b36613-26a9-4472-90fd-916bc95383b6" Oct 30 20:30:05.794800 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:05.794785 4292 scope.go:122] "RemoveContainer" containerID="73946197a6f38ea20b666d2931480aa1af31b7d075f24ef272688de85b0686c4" Oct 30 20:30:05.794987 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:30:05.794968 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-b37684-29889870-x8xw2_odh-ai-gateway-infra(a0b2551f-95d5-4a8a-b6e5-08cd9a4d9572)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-b37684-29889870-x8xw2" podUID="a0b2551f-95d5-4a8a-b6e5-08cd9a4d9572" Oct 30 20:30:05.796422 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:05.796401 4292 generic.go:356] "Generic (PLEG): container finished" podID="57e5a5eb-f9ae-42f6-8c6e-e4336b81c953" containerID="317fa17ca05ebb4a72203415fc083167430b21d87f237a48963a35e91e12c369" exitCode=60 Oct 30 20:30:05.796503 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:05.796468 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889870-zws7m" event={"ID":"57e5a5eb-f9ae-42f6-8c6e-e4336b81c953","Type":"ContainerDied","Data":"317fa17ca05ebb4a72203415fc083167430b21d87f237a48963a35e91e12c369"} Oct 30 20:30:05.796671 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:05.796654 4292 scope.go:122] "RemoveContainer" containerID="317fa17ca05ebb4a72203415fc083167430b21d87f237a48963a35e91e12c369" Oct 30 20:30:05.796875 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:30:05.796852 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-b0a183-29889870-zws7m_odh-ai-gateway-infra(57e5a5eb-f9ae-42f6-8c6e-e4336b81c953)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889870-zws7m" podUID="57e5a5eb-f9ae-42f6-8c6e-e4336b81c953" Oct 30 20:30:05.798390 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:05.798343 4292 scope.go:122] "RemoveContainer" containerID="06fdf3e6346498c43375310aba7f1d9dbf748b12c72b53c6866be414c580c7d3" Oct 30 20:30:05.799467 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:30:05.799435 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-cafbb2-29889870-q7kd5_odh-ai-gateway-infra(062d2841-086a-493c-8bd4-e26a0f3c82ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cafbb2-29889870-q7kd5" podUID="062d2841-086a-493c-8bd4-e26a0f3c82ff" Oct 30 20:30:05.800522 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:05.800507 4292 scope.go:122] "RemoveContainer" containerID="accdef60da516f78e5fcb2005e46c35f2b4ea9a5a332764c53868a940cf5a634" Oct 30 20:30:05.800701 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:30:05.800681 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-64120b-29889870-jps8m_odh-ai-gateway-infra(9d3d5c8c-e915-424c-a674-07c2bfb01a7d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64120b-29889870-jps8m" podUID="9d3d5c8c-e915-424c-a674-07c2bfb01a7d" Oct 30 20:30:05.801968 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:05.801955 4292 scope.go:122] "RemoveContainer" containerID="27e8d2f5aa0d5c44820d38c7327788cd82dd3195deea326508ebd371947ada68" Oct 30 20:30:05.814079 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:05.814058 4292 scope.go:122] "RemoveContainer" containerID="a06babce7bdc0f0cfd88e2e1946bbbedd7fd4dc161a87d46cd7da99c12c5dd73" Oct 30 20:30:06.156281 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:06.156245 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz"] Oct 30 20:30:06.161624 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:06.161598 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz" Oct 30 20:30:06.217130 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:06.216959 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-79c649-openshift-default-dockercfg-5hjb9\"" Oct 30 20:30:06.222594 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:06.222551 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz"] Oct 30 20:30:06.286403 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:06.285340 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c8e8826-ddee-49bf-8bbc-daa688aed93a-workload-certs\") pod \"e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz\" (UID: \"2c8e8826-ddee-49bf-8bbc-daa688aed93a\") " pod="openshift-ingress/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz" Oct 30 20:30:06.286403 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:06.285544 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m929d\" (UniqueName: \"kubernetes.io/projected/2c8e8826-ddee-49bf-8bbc-daa688aed93a-kube-api-access-m929d\") pod \"e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz\" (UID: \"2c8e8826-ddee-49bf-8bbc-daa688aed93a\") " pod="openshift-ingress/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz" Oct 30 20:30:06.286403 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:06.285580 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c8e8826-ddee-49bf-8bbc-daa688aed93a-workload-socket\") pod \"e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz\" (UID: \"2c8e8826-ddee-49bf-8bbc-daa688aed93a\") " pod="openshift-ingress/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz" Oct 30 20:30:06.286403 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:06.285616 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c8e8826-ddee-49bf-8bbc-daa688aed93a-istio-token\") pod \"e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz\" (UID: \"2c8e8826-ddee-49bf-8bbc-daa688aed93a\") " pod="openshift-ingress/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz" Oct 30 20:30:06.286403 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:06.285651 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c8e8826-ddee-49bf-8bbc-daa688aed93a-istio-envoy\") pod \"e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz\" (UID: \"2c8e8826-ddee-49bf-8bbc-daa688aed93a\") " pod="openshift-ingress/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz" Oct 30 20:30:06.286403 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:06.285702 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c8e8826-ddee-49bf-8bbc-daa688aed93a-credential-socket\") pod \"e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz\" (UID: \"2c8e8826-ddee-49bf-8bbc-daa688aed93a\") " pod="openshift-ingress/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz" Oct 30 20:30:06.286403 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:06.285731 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c8e8826-ddee-49bf-8bbc-daa688aed93a-istio-podinfo\") pod \"e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz\" (UID: \"2c8e8826-ddee-49bf-8bbc-daa688aed93a\") " pod="openshift-ingress/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz" Oct 30 20:30:06.286403 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:06.285777 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c8e8826-ddee-49bf-8bbc-daa688aed93a-istiod-ca-cert\") pod \"e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz\" (UID: \"2c8e8826-ddee-49bf-8bbc-daa688aed93a\") " pod="openshift-ingress/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz" Oct 30 20:30:06.286403 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:06.285810 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c8e8826-ddee-49bf-8bbc-daa688aed93a-istio-data\") pod \"e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz\" (UID: \"2c8e8826-ddee-49bf-8bbc-daa688aed93a\") " pod="openshift-ingress/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz" Oct 30 20:30:06.395311 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:06.390111 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m929d\" (UniqueName: \"kubernetes.io/projected/2c8e8826-ddee-49bf-8bbc-daa688aed93a-kube-api-access-m929d\") pod \"e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz\" (UID: \"2c8e8826-ddee-49bf-8bbc-daa688aed93a\") " pod="openshift-ingress/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz" Oct 30 20:30:06.395311 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:06.390179 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c8e8826-ddee-49bf-8bbc-daa688aed93a-workload-socket\") pod \"e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz\" (UID: \"2c8e8826-ddee-49bf-8bbc-daa688aed93a\") " pod="openshift-ingress/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz" Oct 30 20:30:06.395311 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:06.390221 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c8e8826-ddee-49bf-8bbc-daa688aed93a-istio-token\") pod \"e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz\" (UID: \"2c8e8826-ddee-49bf-8bbc-daa688aed93a\") " pod="openshift-ingress/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz" Oct 30 20:30:06.395311 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:06.390259 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c8e8826-ddee-49bf-8bbc-daa688aed93a-istio-envoy\") pod \"e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz\" (UID: \"2c8e8826-ddee-49bf-8bbc-daa688aed93a\") " pod="openshift-ingress/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz" Oct 30 20:30:06.395311 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:06.390320 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c8e8826-ddee-49bf-8bbc-daa688aed93a-credential-socket\") pod \"e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz\" (UID: \"2c8e8826-ddee-49bf-8bbc-daa688aed93a\") " pod="openshift-ingress/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz" Oct 30 20:30:06.395311 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:06.390367 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c8e8826-ddee-49bf-8bbc-daa688aed93a-istio-podinfo\") pod \"e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz\" (UID: \"2c8e8826-ddee-49bf-8bbc-daa688aed93a\") " pod="openshift-ingress/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz" Oct 30 20:30:06.395311 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:06.390422 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c8e8826-ddee-49bf-8bbc-daa688aed93a-istiod-ca-cert\") pod \"e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz\" (UID: \"2c8e8826-ddee-49bf-8bbc-daa688aed93a\") " pod="openshift-ingress/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz" Oct 30 20:30:06.395311 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:06.390458 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c8e8826-ddee-49bf-8bbc-daa688aed93a-istio-data\") pod \"e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz\" (UID: \"2c8e8826-ddee-49bf-8bbc-daa688aed93a\") " pod="openshift-ingress/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz" Oct 30 20:30:06.395311 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:06.390524 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c8e8826-ddee-49bf-8bbc-daa688aed93a-workload-certs\") pod \"e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz\" (UID: \"2c8e8826-ddee-49bf-8bbc-daa688aed93a\") " pod="openshift-ingress/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz" Oct 30 20:30:06.395311 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:06.391029 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c8e8826-ddee-49bf-8bbc-daa688aed93a-workload-certs\") pod \"e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz\" (UID: \"2c8e8826-ddee-49bf-8bbc-daa688aed93a\") " pod="openshift-ingress/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz" Oct 30 20:30:06.395311 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:06.391759 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c8e8826-ddee-49bf-8bbc-daa688aed93a-istiod-ca-cert\") pod \"e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz\" (UID: \"2c8e8826-ddee-49bf-8bbc-daa688aed93a\") " pod="openshift-ingress/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz" Oct 30 20:30:06.395311 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:06.392021 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c8e8826-ddee-49bf-8bbc-daa688aed93a-credential-socket\") pod \"e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz\" (UID: \"2c8e8826-ddee-49bf-8bbc-daa688aed93a\") " pod="openshift-ingress/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz" Oct 30 20:30:06.395311 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:06.392212 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c8e8826-ddee-49bf-8bbc-daa688aed93a-istio-data\") pod \"e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz\" (UID: \"2c8e8826-ddee-49bf-8bbc-daa688aed93a\") " pod="openshift-ingress/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz" Oct 30 20:30:06.395311 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:06.392450 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c8e8826-ddee-49bf-8bbc-daa688aed93a-workload-socket\") pod \"e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz\" (UID: \"2c8e8826-ddee-49bf-8bbc-daa688aed93a\") " pod="openshift-ingress/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz" Oct 30 20:30:06.408192 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:06.406288 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c8e8826-ddee-49bf-8bbc-daa688aed93a-istio-envoy\") pod \"e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz\" (UID: \"2c8e8826-ddee-49bf-8bbc-daa688aed93a\") " pod="openshift-ingress/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz" Oct 30 20:30:06.428400 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:06.428313 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c8e8826-ddee-49bf-8bbc-daa688aed93a-istio-podinfo\") pod \"e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz\" (UID: \"2c8e8826-ddee-49bf-8bbc-daa688aed93a\") " pod="openshift-ingress/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz" Oct 30 20:30:06.453942 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:06.453901 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m929d\" (UniqueName: \"kubernetes.io/projected/2c8e8826-ddee-49bf-8bbc-daa688aed93a-kube-api-access-m929d\") pod \"e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz\" (UID: \"2c8e8826-ddee-49bf-8bbc-daa688aed93a\") " pod="openshift-ingress/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz" Oct 30 20:30:06.454368 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:06.454325 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c8e8826-ddee-49bf-8bbc-daa688aed93a-istio-token\") pod \"e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz\" (UID: \"2c8e8826-ddee-49bf-8bbc-daa688aed93a\") " pod="openshift-ingress/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz" Oct 30 20:30:06.480427 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:06.480399 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz" Oct 30 20:30:06.810905 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:06.810877 4292 scope.go:122] "RemoveContainer" containerID="317fa17ca05ebb4a72203415fc083167430b21d87f237a48963a35e91e12c369" Oct 30 20:30:06.811406 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:30:06.811170 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-b0a183-29889870-zws7m_odh-ai-gateway-infra(57e5a5eb-f9ae-42f6-8c6e-e4336b81c953)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889870-zws7m" podUID="57e5a5eb-f9ae-42f6-8c6e-e4336b81c953" Oct 30 20:30:06.812948 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:06.812928 4292 scope.go:122] "RemoveContainer" containerID="ef42c81409c6ad94344985ba6c22f62035870189adb72e70fca7afe21d9176ee" Oct 30 20:30:06.813212 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:30:06.813189 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-0e72bd-29889870-444qs_odh-ai-gateway-infra(ff6b576c-bc3c-4a15-ae30-57474f6285d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0e72bd-29889870-444qs" podUID="ff6b576c-bc3c-4a15-ae30-57474f6285d0" Oct 30 20:30:06.814886 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:06.814862 4292 scope.go:122] "RemoveContainer" containerID="f36bb8a65af8ad97b7be284fd2b47f5b673abdb6eb2aa6435edc397aa9ba8109" Oct 30 20:30:06.815140 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:30:06.815117 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-487c5f-29889870-pljq2_odh-ai-gateway-infra(32b36613-26a9-4472-90fd-916bc95383b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-487c5f-29889870-pljq2" podUID="32b36613-26a9-4472-90fd-916bc95383b6" Oct 30 20:30:06.914997 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:30:06.914968 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c8e8826_ddee_49bf_8bbc_daa688aed93a.slice/crio-70ac8c916827646e8fb937eec104f97502e7b40962c23b8b71dbade02e96814a WatchSource:0}: Error finding container 70ac8c916827646e8fb937eec104f97502e7b40962c23b8b71dbade02e96814a: Status 404 returned error can't find the container with id 70ac8c916827646e8fb937eec104f97502e7b40962c23b8b71dbade02e96814a Oct 30 20:30:06.918238 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:06.918203 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:30:06.918374 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:06.918274 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz"] Oct 30 20:30:06.918374 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:06.918284 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:30:06.918374 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:06.918331 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:30:07.822696 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:07.822654 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz" event={"ID":"2c8e8826-ddee-49bf-8bbc-daa688aed93a","Type":"ContainerStarted","Data":"f4c4a4283ac98a8492f7b0ae07698ffee400affd9944fdc24894074039d622dc"} Oct 30 20:30:07.823116 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:07.822719 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz" event={"ID":"2c8e8826-ddee-49bf-8bbc-daa688aed93a","Type":"ContainerStarted","Data":"70ac8c916827646e8fb937eec104f97502e7b40962c23b8b71dbade02e96814a"} Oct 30 20:30:07.948711 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:07.948652 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz" podStartSLOduration=1.948633811 podStartE2EDuration="1.948633811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:30:07.944641731 +0000 UTC m=+3911.462957216" watchObservedRunningTime="2026-10-30 20:30:07.948633811 +0000 UTC m=+3911.466949267" Oct 30 20:30:08.173440 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:08.168087 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-64120b/auth-test-model-64120b-kserve-8968677fc-q4rsd"] Oct 30 20:30:08.481261 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:08.481178 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz" Oct 30 20:30:08.487177 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:08.487146 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz" Oct 30 20:30:08.828540 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:08.828516 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz" Oct 30 20:30:08.829677 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:08.829660 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz" Oct 30 20:30:09.836698 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:09.836667 4292 generic.go:356] "Generic (PLEG): container finished" podID="03c3a9f1-5156-41a2-a643-46343f91b78e" containerID="f61b08f353f0f4e288f4c040cd0e9ab5164fe2efc409f8a2bf1c0d2aaecfac32" exitCode=0 Oct 30 20:30:09.837191 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:09.836745 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds" event={"ID":"03c3a9f1-5156-41a2-a643-46343f91b78e","Type":"ContainerDied","Data":"f61b08f353f0f4e288f4c040cd0e9ab5164fe2efc409f8a2bf1c0d2aaecfac32"} Oct 30 20:30:09.837240 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:09.837214 4292 scope.go:122] "RemoveContainer" containerID="f61b08f353f0f4e288f4c040cd0e9ab5164fe2efc409f8a2bf1c0d2aaecfac32" Oct 30 20:30:09.838846 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:09.838815 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:30:09.838967 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:09.838901 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:30:09.838967 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:09.838953 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:30:10.845636 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:10.845556 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds" event={"ID":"03c3a9f1-5156-41a2-a643-46343f91b78e","Type":"ContainerStarted","Data":"ca5f1572b9c2328c499c179600bf48fd0b6f62c20d6f6ea92cac3847e446678d"} Oct 30 20:30:11.130892 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:11.130811 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds" Oct 30 20:30:11.132133 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:11.132074 4292 patch_prober.go:29] interesting pod/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Oct 30 20:30:11.132279 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:11.132174 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds" podUID="03c3a9f1-5156-41a2-a643-46343f91b78e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Oct 30 20:30:12.130882 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:12.130846 4292 patch_prober.go:29] interesting pod/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Oct 30 20:30:12.131291 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:12.130911 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds" podUID="03c3a9f1-5156-41a2-a643-46343f91b78e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Oct 30 20:30:12.905937 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:12.905904 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds" Oct 30 20:30:13.131055 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:13.131015 4292 patch_prober.go:29] interesting pod/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Oct 30 20:30:13.131504 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:13.131117 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds" podUID="03c3a9f1-5156-41a2-a643-46343f91b78e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Oct 30 20:30:13.223111 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:30:13.223040 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:30:14.130928 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:14.130896 4292 patch_prober.go:29] interesting pod/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Oct 30 20:30:14.131127 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:14.130963 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds" podUID="03c3a9f1-5156-41a2-a643-46343f91b78e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Oct 30 20:30:14.586971 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:14.586939 4292 scope.go:122] "RemoveContainer" containerID="accdef60da516f78e5fcb2005e46c35f2b4ea9a5a332764c53868a940cf5a634" Oct 30 20:30:15.130305 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:15.130269 4292 patch_prober.go:29] interesting pod/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Oct 30 20:30:15.130495 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:15.130324 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds" podUID="03c3a9f1-5156-41a2-a643-46343f91b78e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Oct 30 20:30:15.576070 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:15.576040 4292 scope.go:122] "RemoveContainer" containerID="317fa17ca05ebb4a72203415fc083167430b21d87f237a48963a35e91e12c369" Oct 30 20:30:15.923554 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:15.923521 4292 generic.go:356] "Generic (PLEG): container finished" podID="9d3d5c8c-e915-424c-a674-07c2bfb01a7d" containerID="c013490cb32decd3faa8b73f1daa2df3bb5fef2bc82629ae091f559b53d05ba9" exitCode=60 Oct 30 20:30:15.923741 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:15.923594 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64120b-29889870-jps8m" event={"ID":"9d3d5c8c-e915-424c-a674-07c2bfb01a7d","Type":"ContainerDied","Data":"c013490cb32decd3faa8b73f1daa2df3bb5fef2bc82629ae091f559b53d05ba9"} Oct 30 20:30:15.923741 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:15.923651 4292 scope.go:122] "RemoveContainer" containerID="accdef60da516f78e5fcb2005e46c35f2b4ea9a5a332764c53868a940cf5a634" Oct 30 20:30:15.924140 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:15.924123 4292 scope.go:122] "RemoveContainer" containerID="c013490cb32decd3faa8b73f1daa2df3bb5fef2bc82629ae091f559b53d05ba9" Oct 30 20:30:15.924457 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:30:15.924436 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-64120b-29889870-jps8m_odh-ai-gateway-infra(9d3d5c8c-e915-424c-a674-07c2bfb01a7d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64120b-29889870-jps8m" podUID="9d3d5c8c-e915-424c-a674-07c2bfb01a7d" Oct 30 20:30:15.925943 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:15.925922 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889870-zws7m" event={"ID":"57e5a5eb-f9ae-42f6-8c6e-e4336b81c953","Type":"ContainerStarted","Data":"0856452cc29109ce94ee21b5a9e995fd79db3fa13256ce471ef81f1aeafbb1f9"} Oct 30 20:30:16.135423 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:16.135336 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds" Oct 30 20:30:16.136453 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:16.136429 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds" Oct 30 20:30:16.583247 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:16.583218 4292 scope.go:122] "RemoveContainer" containerID="73946197a6f38ea20b666d2931480aa1af31b7d075f24ef272688de85b0686c4" Oct 30 20:30:16.583775 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:16.583328 4292 scope.go:122] "RemoveContainer" containerID="ef42c81409c6ad94344985ba6c22f62035870189adb72e70fca7afe21d9176ee" Oct 30 20:30:16.933620 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:16.933591 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-b37684-29889870-x8xw2" event={"ID":"a0b2551f-95d5-4a8a-b6e5-08cd9a4d9572","Type":"ContainerStarted","Data":"7a9a5825e8e639f97bcfb941eba161e9888225d5b117c5ca6629e33bf34f2334"} Oct 30 20:30:16.935457 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:16.935431 4292 generic.go:356] "Generic (PLEG): container finished" podID="57e5a5eb-f9ae-42f6-8c6e-e4336b81c953" containerID="0856452cc29109ce94ee21b5a9e995fd79db3fa13256ce471ef81f1aeafbb1f9" exitCode=60 Oct 30 20:30:16.935584 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:16.935502 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889870-zws7m" event={"ID":"57e5a5eb-f9ae-42f6-8c6e-e4336b81c953","Type":"ContainerDied","Data":"0856452cc29109ce94ee21b5a9e995fd79db3fa13256ce471ef81f1aeafbb1f9"} Oct 30 20:30:16.935584 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:16.935549 4292 scope.go:122] "RemoveContainer" containerID="317fa17ca05ebb4a72203415fc083167430b21d87f237a48963a35e91e12c369" Oct 30 20:30:16.935867 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:16.935851 4292 scope.go:122] "RemoveContainer" containerID="0856452cc29109ce94ee21b5a9e995fd79db3fa13256ce471ef81f1aeafbb1f9" Oct 30 20:30:16.936159 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:30:16.936138 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-b0a183-29889870-zws7m_odh-ai-gateway-infra(57e5a5eb-f9ae-42f6-8c6e-e4336b81c953)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889870-zws7m" podUID="57e5a5eb-f9ae-42f6-8c6e-e4336b81c953" Oct 30 20:30:17.062233 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:17.062208 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64120b-29889870-jps8m"] Oct 30 20:30:17.185618 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:17.185592 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889870-zws7m"] Oct 30 20:30:17.200266 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:17.200246 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64120b-29889870-jps8m" Oct 30 20:30:17.312067 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:17.311603 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d3d5c8c-e915-424c-a674-07c2bfb01a7d-kube-api-access-6z9wh\" (UniqueName: \"kubernetes.io/projected/9d3d5c8c-e915-424c-a674-07c2bfb01a7d-kube-api-access-6z9wh\") pod \"9d3d5c8c-e915-424c-a674-07c2bfb01a7d\" (UID: \"9d3d5c8c-e915-424c-a674-07c2bfb01a7d\") " Oct 30 20:30:17.312067 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:17.311659 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d3d5c8c-e915-424c-a674-07c2bfb01a7d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d3d5c8c-e915-424c-a674-07c2bfb01a7d-maas-api-service-ca\") pod \"9d3d5c8c-e915-424c-a674-07c2bfb01a7d\" (UID: \"9d3d5c8c-e915-424c-a674-07c2bfb01a7d\") " Oct 30 20:30:17.312371 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:17.312192 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d3d5c8c-e915-424c-a674-07c2bfb01a7d-maas-api-service-ca" pod "9d3d5c8c-e915-424c-a674-07c2bfb01a7d" (UID: "9d3d5c8c-e915-424c-a674-07c2bfb01a7d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:30:17.315621 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:17.315585 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d3d5c8c-e915-424c-a674-07c2bfb01a7d-kube-api-access-6z9wh" pod "9d3d5c8c-e915-424c-a674-07c2bfb01a7d" (UID: "9d3d5c8c-e915-424c-a674-07c2bfb01a7d"). InnerVolumeSpecName "kube-api-access-6z9wh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:30:17.413701 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:17.413633 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z9wh\" (UniqueName: \"kubernetes.io/projected/9d3d5c8c-e915-424c-a674-07c2bfb01a7d-kube-api-access-6z9wh\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:30:17.413701 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:17.413659 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d3d5c8c-e915-424c-a674-07c2bfb01a7d-maas-api-service-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:30:17.575622 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:17.575586 4292 scope.go:122] "RemoveContainer" containerID="06fdf3e6346498c43375310aba7f1d9dbf748b12c72b53c6866be414c580c7d3" Oct 30 20:30:17.575804 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:17.575716 4292 scope.go:122] "RemoveContainer" containerID="85bbb20cdf12173def845e0ef724a039811f740001b5734ffb28430055cf643d" Oct 30 20:30:17.575878 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:17.575835 4292 scope.go:122] "RemoveContainer" containerID="0cc7122047009ca0116eed8552ca372be3e1e8985f5c81bcd03a2fe51880aa19" Oct 30 20:30:17.575930 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:17.575909 4292 scope.go:122] "RemoveContainer" containerID="1db0ade25f1374b29acef8da2c2efc5f28821962b58ac7446b52a09ce7d965bd" Oct 30 20:30:17.944851 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:17.944763 4292 generic.go:356] "Generic (PLEG): container finished" podID="a0b2551f-95d5-4a8a-b6e5-08cd9a4d9572" containerID="7a9a5825e8e639f97bcfb941eba161e9888225d5b117c5ca6629e33bf34f2334" exitCode=60 Oct 30 20:30:17.944851 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:17.944835 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-b37684-29889870-x8xw2" event={"ID":"a0b2551f-95d5-4a8a-b6e5-08cd9a4d9572","Type":"ContainerDied","Data":"7a9a5825e8e639f97bcfb941eba161e9888225d5b117c5ca6629e33bf34f2334"} Oct 30 20:30:17.945424 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:17.944902 4292 scope.go:122] "RemoveContainer" containerID="73946197a6f38ea20b666d2931480aa1af31b7d075f24ef272688de85b0686c4" Oct 30 20:30:17.945424 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:17.945276 4292 scope.go:122] "RemoveContainer" containerID="7a9a5825e8e639f97bcfb941eba161e9888225d5b117c5ca6629e33bf34f2334" Oct 30 20:30:17.945644 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:30:17.945623 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-b37684-29889870-x8xw2_odh-ai-gateway-infra(a0b2551f-95d5-4a8a-b6e5-08cd9a4d9572)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-b37684-29889870-x8xw2" podUID="a0b2551f-95d5-4a8a-b6e5-08cd9a4d9572" Oct 30 20:30:17.950869 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:17.948222 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64120b-29889870-jps8m" Oct 30 20:30:17.950869 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:17.948275 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64120b-29889870-jps8m" event={"ID":"9d3d5c8c-e915-424c-a674-07c2bfb01a7d","Type":"ContainerDied","Data":"83a3d6be4a6e4608ac251829885f16395b69f74258dd181c6b0bd94da37af46b"} Oct 30 20:30:17.950869 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:17.950755 4292 generic.go:356] "Generic (PLEG): container finished" podID="ff6b576c-bc3c-4a15-ae30-57474f6285d0" containerID="7e2c72365c40e91484d1d839bb1e8db053ed01977cca43e7cbc43f015ed5f485" exitCode=60 Oct 30 20:30:17.950869 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:17.950849 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0e72bd-29889870-444qs" event={"ID":"ff6b576c-bc3c-4a15-ae30-57474f6285d0","Type":"ContainerDied","Data":"7e2c72365c40e91484d1d839bb1e8db053ed01977cca43e7cbc43f015ed5f485"} Oct 30 20:30:17.951531 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:17.951508 4292 scope.go:122] "RemoveContainer" containerID="7e2c72365c40e91484d1d839bb1e8db053ed01977cca43e7cbc43f015ed5f485" Oct 30 20:30:17.951827 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:30:17.951803 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-0e72bd-29889870-444qs_odh-ai-gateway-infra(ff6b576c-bc3c-4a15-ae30-57474f6285d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0e72bd-29889870-444qs" podUID="ff6b576c-bc3c-4a15-ae30-57474f6285d0" Oct 30 20:30:17.961206 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:17.961189 4292 scope.go:122] "RemoveContainer" containerID="c013490cb32decd3faa8b73f1daa2df3bb5fef2bc82629ae091f559b53d05ba9" Oct 30 20:30:17.978523 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:17.978503 4292 scope.go:122] "RemoveContainer" containerID="ef42c81409c6ad94344985ba6c22f62035870189adb72e70fca7afe21d9176ee" Oct 30 20:30:18.071018 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:18.070995 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-b37684-29889870-x8xw2"] Oct 30 20:30:18.075901 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:18.075885 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889870-zws7m" Oct 30 20:30:18.179844 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:18.179822 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64120b-29889870-jps8m"] Oct 30 20:30:18.184704 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:18.184665 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64120b-29889870-jps8m"] Oct 30 20:30:18.223700 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:18.223644 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57e5a5eb-f9ae-42f6-8c6e-e4336b81c953-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57e5a5eb-f9ae-42f6-8c6e-e4336b81c953-maas-api-service-ca\") pod \"57e5a5eb-f9ae-42f6-8c6e-e4336b81c953\" (UID: \"57e5a5eb-f9ae-42f6-8c6e-e4336b81c953\") " Oct 30 20:30:18.223803 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:18.223751 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57e5a5eb-f9ae-42f6-8c6e-e4336b81c953-kube-api-access-b6bqv\" (UniqueName: \"kubernetes.io/projected/57e5a5eb-f9ae-42f6-8c6e-e4336b81c953-kube-api-access-b6bqv\") pod \"57e5a5eb-f9ae-42f6-8c6e-e4336b81c953\" (UID: \"57e5a5eb-f9ae-42f6-8c6e-e4336b81c953\") " Oct 30 20:30:18.224014 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:18.223989 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57e5a5eb-f9ae-42f6-8c6e-e4336b81c953-maas-api-service-ca" pod "57e5a5eb-f9ae-42f6-8c6e-e4336b81c953" (UID: "57e5a5eb-f9ae-42f6-8c6e-e4336b81c953"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:30:18.225609 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:18.225590 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57e5a5eb-f9ae-42f6-8c6e-e4336b81c953-kube-api-access-b6bqv" pod "57e5a5eb-f9ae-42f6-8c6e-e4336b81c953" (UID: "57e5a5eb-f9ae-42f6-8c6e-e4336b81c953"). InnerVolumeSpecName "kube-api-access-b6bqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:30:18.324742 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:18.324714 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57e5a5eb-f9ae-42f6-8c6e-e4336b81c953-maas-api-service-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:30:18.324742 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:18.324738 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6bqv\" (UniqueName: \"kubernetes.io/projected/57e5a5eb-f9ae-42f6-8c6e-e4336b81c953-kube-api-access-b6bqv\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:30:18.594180 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:18.594142 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d3d5c8c-e915-424c-a674-07c2bfb01a7d" path="/var/lib/kubelet/pods/9d3d5c8c-e915-424c-a674-07c2bfb01a7d/volumes" Oct 30 20:30:18.957757 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:18.957725 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889870-zws7m" event={"ID":"57e5a5eb-f9ae-42f6-8c6e-e4336b81c953","Type":"ContainerDied","Data":"cdb3213417ce95e5a439cc59d48449d9cfafd2d1b45cf2a9104b845107edd41c"} Oct 30 20:30:18.957757 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:18.957751 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889870-zws7m" Oct 30 20:30:18.958248 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:18.957791 4292 scope.go:122] "RemoveContainer" containerID="0856452cc29109ce94ee21b5a9e995fd79db3fa13256ce471ef81f1aeafbb1f9" Oct 30 20:30:18.960033 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:18.960008 4292 generic.go:356] "Generic (PLEG): container finished" podID="062d2841-086a-493c-8bd4-e26a0f3c82ff" containerID="0cc95a8f37a46319af9d0cb2bd06a47667bb779070f37f5e8069950fed8b7799" exitCode=60 Oct 30 20:30:18.960181 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:18.960093 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cafbb2-29889870-q7kd5" event={"ID":"062d2841-086a-493c-8bd4-e26a0f3c82ff","Type":"ContainerDied","Data":"0cc95a8f37a46319af9d0cb2bd06a47667bb779070f37f5e8069950fed8b7799"} Oct 30 20:30:18.960622 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:18.960598 4292 scope.go:122] "RemoveContainer" containerID="0cc95a8f37a46319af9d0cb2bd06a47667bb779070f37f5e8069950fed8b7799" Oct 30 20:30:18.960936 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:30:18.960910 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-cafbb2-29889870-q7kd5_odh-ai-gateway-infra(062d2841-086a-493c-8bd4-e26a0f3c82ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cafbb2-29889870-q7kd5" podUID="062d2841-086a-493c-8bd4-e26a0f3c82ff" Oct 30 20:30:18.963101 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:18.963082 4292 generic.go:356] "Generic (PLEG): container finished" podID="bc4d4a92-2ef3-42a5-82be-70bb892011b1" containerID="52a9476c7d4d2fa380c14fcfe55fd12b6960f85063f22efe2eddb4771ff30586" exitCode=60 Oct 30 20:30:18.963208 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:18.963156 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-f6rv5" event={"ID":"bc4d4a92-2ef3-42a5-82be-70bb892011b1","Type":"ContainerDied","Data":"52a9476c7d4d2fa380c14fcfe55fd12b6960f85063f22efe2eddb4771ff30586"} Oct 30 20:30:18.963572 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:18.963552 4292 scope.go:122] "RemoveContainer" containerID="52a9476c7d4d2fa380c14fcfe55fd12b6960f85063f22efe2eddb4771ff30586" Oct 30 20:30:18.963829 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:30:18.963811 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889870-f6rv5_odh-ai-gateway-infra(bc4d4a92-2ef3-42a5-82be-70bb892011b1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-f6rv5" podUID="bc4d4a92-2ef3-42a5-82be-70bb892011b1" Oct 30 20:30:18.964929 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:18.964909 4292 generic.go:356] "Generic (PLEG): container finished" podID="a895dc65-411f-468e-8872-3fb6b6e35a5e" containerID="565671c84cfcc6f9176b760f9c1e57fcff55c843e3180018ca749bdb50c9925a" exitCode=60 Oct 30 20:30:18.965004 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:18.964979 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e4864d-29889870-djv4g" event={"ID":"a895dc65-411f-468e-8872-3fb6b6e35a5e","Type":"ContainerDied","Data":"565671c84cfcc6f9176b760f9c1e57fcff55c843e3180018ca749bdb50c9925a"} Oct 30 20:30:18.965310 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:18.965293 4292 scope.go:122] "RemoveContainer" containerID="565671c84cfcc6f9176b760f9c1e57fcff55c843e3180018ca749bdb50c9925a" Oct 30 20:30:18.965593 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:30:18.965575 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-e4864d-29889870-djv4g_odh-ai-gateway-infra(a895dc65-411f-468e-8872-3fb6b6e35a5e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e4864d-29889870-djv4g" podUID="a895dc65-411f-468e-8872-3fb6b6e35a5e" Oct 30 20:30:18.968272 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:18.968252 4292 generic.go:356] "Generic (PLEG): container finished" podID="79ffe45a-2987-49cc-909c-920ccbbbb53a" containerID="a01887e36b700f833a5fc4171de13c9b019afaf5f6e86f34d8c8f39b7c5c3483" exitCode=60 Oct 30 20:30:18.968386 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:18.968326 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-056a27-29889870-4tb2q" event={"ID":"79ffe45a-2987-49cc-909c-920ccbbbb53a","Type":"ContainerDied","Data":"a01887e36b700f833a5fc4171de13c9b019afaf5f6e86f34d8c8f39b7c5c3483"} Oct 30 20:30:18.968537 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:18.968521 4292 scope.go:122] "RemoveContainer" containerID="06fdf3e6346498c43375310aba7f1d9dbf748b12c72b53c6866be414c580c7d3" Oct 30 20:30:18.968706 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:18.968678 4292 scope.go:122] "RemoveContainer" containerID="a01887e36b700f833a5fc4171de13c9b019afaf5f6e86f34d8c8f39b7c5c3483" Oct 30 20:30:18.969035 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:30:18.968907 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-056a27-29889870-4tb2q_odh-ai-gateway-infra(79ffe45a-2987-49cc-909c-920ccbbbb53a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-056a27-29889870-4tb2q" podUID="79ffe45a-2987-49cc-909c-920ccbbbb53a" Oct 30 20:30:18.979813 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:18.979794 4292 scope.go:122] "RemoveContainer" containerID="85bbb20cdf12173def845e0ef724a039811f740001b5734ffb28430055cf643d" Oct 30 20:30:18.991022 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:18.991002 4292 scope.go:122] "RemoveContainer" containerID="0cc7122047009ca0116eed8552ca372be3e1e8985f5c81bcd03a2fe51880aa19" Oct 30 20:30:19.010141 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:19.010121 4292 scope.go:122] "RemoveContainer" containerID="1db0ade25f1374b29acef8da2c2efc5f28821962b58ac7446b52a09ce7d965bd" Oct 30 20:30:19.109107 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:19.109082 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-b37684-29889870-x8xw2" Oct 30 20:30:19.234470 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:19.234438 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0b2551f-95d5-4a8a-b6e5-08cd9a4d9572-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b2551f-95d5-4a8a-b6e5-08cd9a4d9572-maas-api-service-ca\") pod \"a0b2551f-95d5-4a8a-b6e5-08cd9a4d9572\" (UID: \"a0b2551f-95d5-4a8a-b6e5-08cd9a4d9572\") " Oct 30 20:30:19.234661 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:19.234501 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0b2551f-95d5-4a8a-b6e5-08cd9a4d9572-kube-api-access-7nkff\" (UniqueName: \"kubernetes.io/projected/a0b2551f-95d5-4a8a-b6e5-08cd9a4d9572-kube-api-access-7nkff\") pod \"a0b2551f-95d5-4a8a-b6e5-08cd9a4d9572\" (UID: \"a0b2551f-95d5-4a8a-b6e5-08cd9a4d9572\") " Oct 30 20:30:19.234818 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:19.234799 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0b2551f-95d5-4a8a-b6e5-08cd9a4d9572-maas-api-service-ca" pod "a0b2551f-95d5-4a8a-b6e5-08cd9a4d9572" (UID: "a0b2551f-95d5-4a8a-b6e5-08cd9a4d9572"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:30:19.236479 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:19.236460 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0b2551f-95d5-4a8a-b6e5-08cd9a4d9572-kube-api-access-7nkff" pod "a0b2551f-95d5-4a8a-b6e5-08cd9a4d9572" (UID: "a0b2551f-95d5-4a8a-b6e5-08cd9a4d9572"). InnerVolumeSpecName "kube-api-access-7nkff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:30:19.250446 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:19.250418 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889870-zws7m"] Oct 30 20:30:19.251389 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:19.251370 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889870-zws7m"] Oct 30 20:30:19.311955 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:19.311924 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0e72bd-29889870-444qs"] Oct 30 20:30:19.335849 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:19.335815 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b2551f-95d5-4a8a-b6e5-08cd9a4d9572-maas-api-service-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:30:19.335849 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:19.335842 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nkff\" (UniqueName: \"kubernetes.io/projected/a0b2551f-95d5-4a8a-b6e5-08cd9a4d9572-kube-api-access-7nkff\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:30:19.445431 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:19.445409 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0e72bd-29889870-444qs" Oct 30 20:30:19.539390 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:19.539335 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff6b576c-bc3c-4a15-ae30-57474f6285d0-kube-api-access-sctzl\" (UniqueName: \"kubernetes.io/projected/ff6b576c-bc3c-4a15-ae30-57474f6285d0-kube-api-access-sctzl\") pod \"ff6b576c-bc3c-4a15-ae30-57474f6285d0\" (UID: \"ff6b576c-bc3c-4a15-ae30-57474f6285d0\") " Oct 30 20:30:19.539564 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:19.539501 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff6b576c-bc3c-4a15-ae30-57474f6285d0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff6b576c-bc3c-4a15-ae30-57474f6285d0-maas-api-service-ca\") pod \"ff6b576c-bc3c-4a15-ae30-57474f6285d0\" (UID: \"ff6b576c-bc3c-4a15-ae30-57474f6285d0\") " Oct 30 20:30:19.539867 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:19.539845 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff6b576c-bc3c-4a15-ae30-57474f6285d0-maas-api-service-ca" pod "ff6b576c-bc3c-4a15-ae30-57474f6285d0" (UID: "ff6b576c-bc3c-4a15-ae30-57474f6285d0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:30:19.540012 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:19.539995 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff6b576c-bc3c-4a15-ae30-57474f6285d0-maas-api-service-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:30:19.541653 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:19.541633 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff6b576c-bc3c-4a15-ae30-57474f6285d0-kube-api-access-sctzl" pod "ff6b576c-bc3c-4a15-ae30-57474f6285d0" (UID: "ff6b576c-bc3c-4a15-ae30-57474f6285d0"). InnerVolumeSpecName "kube-api-access-sctzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:30:19.576126 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:19.576107 4292 scope.go:122] "RemoveContainer" containerID="f36bb8a65af8ad97b7be284fd2b47f5b673abdb6eb2aa6435edc397aa9ba8109" Oct 30 20:30:19.641401 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:19.641293 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sctzl\" (UniqueName: \"kubernetes.io/projected/ff6b576c-bc3c-4a15-ae30-57474f6285d0-kube-api-access-sctzl\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:30:19.979416 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:19.979319 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0e72bd-29889870-444qs" Oct 30 20:30:19.979416 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:19.979398 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0e72bd-29889870-444qs" event={"ID":"ff6b576c-bc3c-4a15-ae30-57474f6285d0","Type":"ContainerDied","Data":"d75a17a892e7c5917fb693b6f51b28e40b85fe24b33225e5b76c2f445406e66f"} Oct 30 20:30:19.979937 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:19.979468 4292 scope.go:122] "RemoveContainer" containerID="7e2c72365c40e91484d1d839bb1e8db053ed01977cca43e7cbc43f015ed5f485" Oct 30 20:30:19.983962 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:19.983934 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-487c5f-29889870-pljq2" event={"ID":"32b36613-26a9-4472-90fd-916bc95383b6","Type":"ContainerStarted","Data":"2fc2529effa53e624775d3dabf004d37d6131c9e30c84918daae62e2fc4c291b"} Oct 30 20:30:19.987264 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:19.987237 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-b37684-29889870-x8xw2" event={"ID":"a0b2551f-95d5-4a8a-b6e5-08cd9a4d9572","Type":"ContainerDied","Data":"396028c7442f124256b5378a20921db1856ec3131f629162eec59c7c17d2869a"} Oct 30 20:30:19.987421 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:19.987316 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-b37684-29889870-x8xw2" Oct 30 20:30:19.997440 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:19.992974 4292 scope.go:122] "RemoveContainer" containerID="7a9a5825e8e639f97bcfb941eba161e9888225d5b117c5ca6629e33bf34f2334" Oct 30 20:30:20.162467 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:20.162430 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-b37684-29889870-x8xw2"] Oct 30 20:30:20.166184 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:20.166159 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-b37684-29889870-x8xw2"] Oct 30 20:30:20.384735 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:20.384700 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e4864d-29889870-djv4g"] Oct 30 20:30:20.399826 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:20.399796 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0e72bd-29889870-444qs"] Oct 30 20:30:20.410026 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:20.409985 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0e72bd-29889870-444qs"] Oct 30 20:30:20.506370 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:20.502648 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889870-f6rv5"] Oct 30 20:30:20.586569 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:20.586541 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57e5a5eb-f9ae-42f6-8c6e-e4336b81c953" path="/var/lib/kubelet/pods/57e5a5eb-f9ae-42f6-8c6e-e4336b81c953/volumes" Oct 30 20:30:20.587078 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:20.587058 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0b2551f-95d5-4a8a-b6e5-08cd9a4d9572" path="/var/lib/kubelet/pods/a0b2551f-95d5-4a8a-b6e5-08cd9a4d9572/volumes" Oct 30 20:30:20.587534 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:20.587516 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff6b576c-bc3c-4a15-ae30-57474f6285d0" path="/var/lib/kubelet/pods/ff6b576c-bc3c-4a15-ae30-57474f6285d0/volumes" Oct 30 20:30:20.620778 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:20.620747 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-056a27-29889870-4tb2q"] Oct 30 20:30:20.634610 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:20.634583 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e4864d-29889870-djv4g" Oct 30 20:30:20.701985 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:20.701959 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-f6rv5" Oct 30 20:30:20.737766 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:20.737733 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cafbb2-29889870-q7kd5"] Oct 30 20:30:20.753675 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:20.753646 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a895dc65-411f-468e-8872-3fb6b6e35a5e-kube-api-access-tpgkg\" (UniqueName: \"kubernetes.io/projected/a895dc65-411f-468e-8872-3fb6b6e35a5e-kube-api-access-tpgkg\") pod \"a895dc65-411f-468e-8872-3fb6b6e35a5e\" (UID: \"a895dc65-411f-468e-8872-3fb6b6e35a5e\") " Oct 30 20:30:20.753878 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:20.753744 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a895dc65-411f-468e-8872-3fb6b6e35a5e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a895dc65-411f-468e-8872-3fb6b6e35a5e-maas-api-service-ca\") pod \"a895dc65-411f-468e-8872-3fb6b6e35a5e\" (UID: \"a895dc65-411f-468e-8872-3fb6b6e35a5e\") " Oct 30 20:30:20.754161 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:20.754128 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a895dc65-411f-468e-8872-3fb6b6e35a5e-maas-api-service-ca" pod "a895dc65-411f-468e-8872-3fb6b6e35a5e" (UID: "a895dc65-411f-468e-8872-3fb6b6e35a5e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:30:20.756374 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:20.756328 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a895dc65-411f-468e-8872-3fb6b6e35a5e-kube-api-access-tpgkg" pod "a895dc65-411f-468e-8872-3fb6b6e35a5e" (UID: "a895dc65-411f-468e-8872-3fb6b6e35a5e"). InnerVolumeSpecName "kube-api-access-tpgkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:30:20.780443 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:20.780416 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-056a27-29889870-4tb2q" Oct 30 20:30:20.855409 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:20.855336 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79ffe45a-2987-49cc-909c-920ccbbbb53a-kube-api-access-phmbw\" (UniqueName: \"kubernetes.io/projected/79ffe45a-2987-49cc-909c-920ccbbbb53a-kube-api-access-phmbw\") pod \"79ffe45a-2987-49cc-909c-920ccbbbb53a\" (UID: \"79ffe45a-2987-49cc-909c-920ccbbbb53a\") " Oct 30 20:30:20.855573 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:20.855438 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc4d4a92-2ef3-42a5-82be-70bb892011b1-kube-api-access-8xmpt\" (UniqueName: \"kubernetes.io/projected/bc4d4a92-2ef3-42a5-82be-70bb892011b1-kube-api-access-8xmpt\") pod \"bc4d4a92-2ef3-42a5-82be-70bb892011b1\" (UID: \"bc4d4a92-2ef3-42a5-82be-70bb892011b1\") " Oct 30 20:30:20.855573 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:20.855545 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc4d4a92-2ef3-42a5-82be-70bb892011b1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc4d4a92-2ef3-42a5-82be-70bb892011b1-maas-api-service-ca\") pod \"bc4d4a92-2ef3-42a5-82be-70bb892011b1\" (UID: \"bc4d4a92-2ef3-42a5-82be-70bb892011b1\") " Oct 30 20:30:20.855717 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:20.855699 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79ffe45a-2987-49cc-909c-920ccbbbb53a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79ffe45a-2987-49cc-909c-920ccbbbb53a-maas-api-service-ca\") pod \"79ffe45a-2987-49cc-909c-920ccbbbb53a\" (UID: \"79ffe45a-2987-49cc-909c-920ccbbbb53a\") " Oct 30 20:30:20.856067 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:20.856038 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc4d4a92-2ef3-42a5-82be-70bb892011b1-maas-api-service-ca" pod "bc4d4a92-2ef3-42a5-82be-70bb892011b1" (UID: "bc4d4a92-2ef3-42a5-82be-70bb892011b1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:30:20.856186 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:20.856131 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a895dc65-411f-468e-8872-3fb6b6e35a5e-maas-api-service-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:30:20.856186 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:20.856152 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc4d4a92-2ef3-42a5-82be-70bb892011b1-maas-api-service-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:30:20.856186 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:20.856167 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpgkg\" (UniqueName: \"kubernetes.io/projected/a895dc65-411f-468e-8872-3fb6b6e35a5e-kube-api-access-tpgkg\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:30:20.856387 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:20.856296 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79ffe45a-2987-49cc-909c-920ccbbbb53a-maas-api-service-ca" pod "79ffe45a-2987-49cc-909c-920ccbbbb53a" (UID: "79ffe45a-2987-49cc-909c-920ccbbbb53a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:30:20.858605 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:20.858568 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79ffe45a-2987-49cc-909c-920ccbbbb53a-kube-api-access-phmbw" pod "79ffe45a-2987-49cc-909c-920ccbbbb53a" (UID: "79ffe45a-2987-49cc-909c-920ccbbbb53a"). InnerVolumeSpecName "kube-api-access-phmbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:30:20.858711 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:20.858616 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc4d4a92-2ef3-42a5-82be-70bb892011b1-kube-api-access-8xmpt" pod "bc4d4a92-2ef3-42a5-82be-70bb892011b1" (UID: "bc4d4a92-2ef3-42a5-82be-70bb892011b1"). InnerVolumeSpecName "kube-api-access-8xmpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:30:20.892870 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:20.892760 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cafbb2-29889870-q7kd5" Oct 30 20:30:20.957462 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:20.957392 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xmpt\" (UniqueName: \"kubernetes.io/projected/bc4d4a92-2ef3-42a5-82be-70bb892011b1-kube-api-access-8xmpt\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:30:20.957462 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:20.957418 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79ffe45a-2987-49cc-909c-920ccbbbb53a-maas-api-service-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:30:20.957462 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:20.957430 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phmbw\" (UniqueName: \"kubernetes.io/projected/79ffe45a-2987-49cc-909c-920ccbbbb53a-kube-api-access-phmbw\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:30:21.006894 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:21.006867 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cafbb2-29889870-q7kd5" Oct 30 20:30:21.007310 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:21.006893 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cafbb2-29889870-q7kd5" event={"ID":"062d2841-086a-493c-8bd4-e26a0f3c82ff","Type":"ContainerDied","Data":"7a842cd168c192deff7811a15834099408ab085421a9fe990e6160677ae85ffa"} Oct 30 20:30:21.007310 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:21.007002 4292 scope.go:122] "RemoveContainer" containerID="0cc95a8f37a46319af9d0cb2bd06a47667bb779070f37f5e8069950fed8b7799" Oct 30 20:30:21.008780 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:21.008753 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-f6rv5" event={"ID":"bc4d4a92-2ef3-42a5-82be-70bb892011b1","Type":"ContainerDied","Data":"010cc63aedb0d3aad1df9295e084b2f3cb9536c8dd96a304ad3d9b0a882c62f0"} Oct 30 20:30:21.008954 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:21.008788 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-f6rv5" Oct 30 20:30:21.019764 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:21.019293 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e4864d-29889870-djv4g" event={"ID":"a895dc65-411f-468e-8872-3fb6b6e35a5e","Type":"ContainerDied","Data":"8ccc48c04e4a0f1f04747d5e0da0545d19d2a30df12dccc1f17d196407c37dc5"} Oct 30 20:30:21.019764 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:21.019465 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e4864d-29889870-djv4g" Oct 30 20:30:21.020629 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:21.020605 4292 scope.go:122] "RemoveContainer" containerID="52a9476c7d4d2fa380c14fcfe55fd12b6960f85063f22efe2eddb4771ff30586" Oct 30 20:30:21.023490 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:21.023450 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-056a27-29889870-4tb2q" Oct 30 20:30:21.023490 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:21.023460 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-056a27-29889870-4tb2q" event={"ID":"79ffe45a-2987-49cc-909c-920ccbbbb53a","Type":"ContainerDied","Data":"32a0697b187c95a1256569cb1885b45a16a1dd4ce512463c38d39af62d58fc3c"} Oct 30 20:30:21.029147 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:21.029116 4292 generic.go:356] "Generic (PLEG): container finished" podID="32b36613-26a9-4472-90fd-916bc95383b6" containerID="2fc2529effa53e624775d3dabf004d37d6131c9e30c84918daae62e2fc4c291b" exitCode=60 Oct 30 20:30:21.029282 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:21.029242 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-487c5f-29889870-pljq2" event={"ID":"32b36613-26a9-4472-90fd-916bc95383b6","Type":"ContainerDied","Data":"2fc2529effa53e624775d3dabf004d37d6131c9e30c84918daae62e2fc4c291b"} Oct 30 20:30:21.029748 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:21.029723 4292 scope.go:122] "RemoveContainer" containerID="2fc2529effa53e624775d3dabf004d37d6131c9e30c84918daae62e2fc4c291b" Oct 30 20:30:21.030057 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:30:21.030032 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-487c5f-29889870-pljq2_odh-ai-gateway-infra(32b36613-26a9-4472-90fd-916bc95383b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-487c5f-29889870-pljq2" podUID="32b36613-26a9-4472-90fd-916bc95383b6" Oct 30 20:30:21.038483 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:21.038463 4292 scope.go:122] "RemoveContainer" containerID="565671c84cfcc6f9176b760f9c1e57fcff55c843e3180018ca749bdb50c9925a" Oct 30 20:30:21.047681 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:21.047658 4292 scope.go:122] "RemoveContainer" containerID="a01887e36b700f833a5fc4171de13c9b019afaf5f6e86f34d8c8f39b7c5c3483" Oct 30 20:30:21.056111 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:21.056095 4292 scope.go:122] "RemoveContainer" containerID="f36bb8a65af8ad97b7be284fd2b47f5b673abdb6eb2aa6435edc397aa9ba8109" Oct 30 20:30:21.058241 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:21.058219 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/062d2841-086a-493c-8bd4-e26a0f3c82ff-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/062d2841-086a-493c-8bd4-e26a0f3c82ff-maas-api-service-ca\") pod \"062d2841-086a-493c-8bd4-e26a0f3c82ff\" (UID: \"062d2841-086a-493c-8bd4-e26a0f3c82ff\") " Oct 30 20:30:21.058344 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:21.058290 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/062d2841-086a-493c-8bd4-e26a0f3c82ff-kube-api-access-s8rgk\" (UniqueName: \"kubernetes.io/projected/062d2841-086a-493c-8bd4-e26a0f3c82ff-kube-api-access-s8rgk\") pod \"062d2841-086a-493c-8bd4-e26a0f3c82ff\" (UID: \"062d2841-086a-493c-8bd4-e26a0f3c82ff\") " Oct 30 20:30:21.058543 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:21.058523 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/062d2841-086a-493c-8bd4-e26a0f3c82ff-maas-api-service-ca" pod "062d2841-086a-493c-8bd4-e26a0f3c82ff" (UID: "062d2841-086a-493c-8bd4-e26a0f3c82ff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:30:21.058773 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:21.058755 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/062d2841-086a-493c-8bd4-e26a0f3c82ff-maas-api-service-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:30:21.060296 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:21.060273 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/062d2841-086a-493c-8bd4-e26a0f3c82ff-kube-api-access-s8rgk" pod "062d2841-086a-493c-8bd4-e26a0f3c82ff" (UID: "062d2841-086a-493c-8bd4-e26a0f3c82ff"). InnerVolumeSpecName "kube-api-access-s8rgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:30:21.159478 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:21.159444 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8rgk\" (UniqueName: \"kubernetes.io/projected/062d2841-086a-493c-8bd4-e26a0f3c82ff-kube-api-access-s8rgk\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:30:21.298896 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:21.298834 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-487c5f-29889870-pljq2"] Oct 30 20:30:21.372774 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:21.372745 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e4864d-29889870-djv4g"] Oct 30 20:30:21.378038 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:21.378012 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e4864d-29889870-djv4g"] Oct 30 20:30:21.607989 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:21.607958 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889870-f6rv5"] Oct 30 20:30:21.612872 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:21.612844 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889870-f6rv5"] Oct 30 20:30:21.749876 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:21.749843 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-056a27-29889870-4tb2q"] Oct 30 20:30:21.764233 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:21.764203 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-056a27-29889870-4tb2q"] Oct 30 20:30:21.992539 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:21.992459 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cafbb2-29889870-q7kd5"] Oct 30 20:30:21.996805 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:21.996780 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cafbb2-29889870-q7kd5"] Oct 30 20:30:22.177698 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:22.177673 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-487c5f-29889870-pljq2" Oct 30 20:30:22.271971 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:22.271947 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32b36613-26a9-4472-90fd-916bc95383b6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32b36613-26a9-4472-90fd-916bc95383b6-maas-api-service-ca\") pod \"32b36613-26a9-4472-90fd-916bc95383b6\" (UID: \"32b36613-26a9-4472-90fd-916bc95383b6\") " Oct 30 20:30:22.272142 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:22.272078 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32b36613-26a9-4472-90fd-916bc95383b6-kube-api-access-6g872\" (UniqueName: \"kubernetes.io/projected/32b36613-26a9-4472-90fd-916bc95383b6-kube-api-access-6g872\") pod \"32b36613-26a9-4472-90fd-916bc95383b6\" (UID: \"32b36613-26a9-4472-90fd-916bc95383b6\") " Oct 30 20:30:22.272334 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:22.272312 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32b36613-26a9-4472-90fd-916bc95383b6-maas-api-service-ca" pod "32b36613-26a9-4472-90fd-916bc95383b6" (UID: "32b36613-26a9-4472-90fd-916bc95383b6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:30:22.274091 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:22.274073 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32b36613-26a9-4472-90fd-916bc95383b6-kube-api-access-6g872" pod "32b36613-26a9-4472-90fd-916bc95383b6" (UID: "32b36613-26a9-4472-90fd-916bc95383b6"). InnerVolumeSpecName "kube-api-access-6g872". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:30:22.373807 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:22.373775 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g872\" (UniqueName: \"kubernetes.io/projected/32b36613-26a9-4472-90fd-916bc95383b6-kube-api-access-6g872\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:30:22.373807 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:22.373806 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32b36613-26a9-4472-90fd-916bc95383b6-maas-api-service-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:30:22.582020 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:22.581948 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="062d2841-086a-493c-8bd4-e26a0f3c82ff" path="/var/lib/kubelet/pods/062d2841-086a-493c-8bd4-e26a0f3c82ff/volumes" Oct 30 20:30:22.582377 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:22.582346 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79ffe45a-2987-49cc-909c-920ccbbbb53a" path="/var/lib/kubelet/pods/79ffe45a-2987-49cc-909c-920ccbbbb53a/volumes" Oct 30 20:30:22.582749 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:22.582737 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a895dc65-411f-468e-8872-3fb6b6e35a5e" path="/var/lib/kubelet/pods/a895dc65-411f-468e-8872-3fb6b6e35a5e/volumes" Oct 30 20:30:22.583385 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:22.583370 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc4d4a92-2ef3-42a5-82be-70bb892011b1" path="/var/lib/kubelet/pods/bc4d4a92-2ef3-42a5-82be-70bb892011b1/volumes" Oct 30 20:30:23.058064 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:23.058033 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-487c5f-29889870-pljq2" Oct 30 20:30:23.058243 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:23.058058 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-487c5f-29889870-pljq2" event={"ID":"32b36613-26a9-4472-90fd-916bc95383b6","Type":"ContainerDied","Data":"52b95354e163a857ea85994f7135795ae256ef3e73ce7835b425cb3d7349da1e"} Oct 30 20:30:23.058243 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:23.058153 4292 scope.go:122] "RemoveContainer" containerID="2fc2529effa53e624775d3dabf004d37d6131c9e30c84918daae62e2fc4c291b" Oct 30 20:30:23.201582 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:23.201520 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-64120b/auth-test-model-64120b-kserve-8968677fc-q4rsd" podUID="8f9c4f3a-c93c-4315-b0c0-95131a3413b6" containerName="main" containerID="cri-o://f2e8ca27df0a0687d642865898b7ee25281a80fae017bc184fc50577dd095cd9" gracePeriod=45 Oct 30 20:30:23.227806 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:23.227781 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-487c5f-29889870-pljq2"] Oct 30 20:30:23.278402 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:23.278368 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-487c5f-29889870-pljq2"] Oct 30 20:30:23.461880 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:23.461857 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-64120b/auth-test-model-64120b-kserve-8968677fc-q4rsd" Oct 30 20:30:23.587578 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:23.587503 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f9c4f3a-c93c-4315-b0c0-95131a3413b6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f9c4f3a-c93c-4315-b0c0-95131a3413b6-dshm\") pod \"8f9c4f3a-c93c-4315-b0c0-95131a3413b6\" (UID: \"8f9c4f3a-c93c-4315-b0c0-95131a3413b6\") " Oct 30 20:30:23.587724 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:23.587589 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f9c4f3a-c93c-4315-b0c0-95131a3413b6-kube-api-access-4ltwn\" (UniqueName: \"kubernetes.io/projected/8f9c4f3a-c93c-4315-b0c0-95131a3413b6-kube-api-access-4ltwn\") pod \"8f9c4f3a-c93c-4315-b0c0-95131a3413b6\" (UID: \"8f9c4f3a-c93c-4315-b0c0-95131a3413b6\") " Oct 30 20:30:23.587724 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:23.587647 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f9c4f3a-c93c-4315-b0c0-95131a3413b6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f9c4f3a-c93c-4315-b0c0-95131a3413b6-tmp-dir\") pod \"8f9c4f3a-c93c-4315-b0c0-95131a3413b6\" (UID: \"8f9c4f3a-c93c-4315-b0c0-95131a3413b6\") " Oct 30 20:30:23.587724 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:23.587695 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f9c4f3a-c93c-4315-b0c0-95131a3413b6-tls-certs\" (UniqueName: \"kubernetes.io/secret/8f9c4f3a-c93c-4315-b0c0-95131a3413b6-tls-certs\") pod \"8f9c4f3a-c93c-4315-b0c0-95131a3413b6\" (UID: \"8f9c4f3a-c93c-4315-b0c0-95131a3413b6\") " Oct 30 20:30:23.587826 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:23.587794 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f9c4f3a-c93c-4315-b0c0-95131a3413b6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f9c4f3a-c93c-4315-b0c0-95131a3413b6-model-cache\") pod \"8f9c4f3a-c93c-4315-b0c0-95131a3413b6\" (UID: \"8f9c4f3a-c93c-4315-b0c0-95131a3413b6\") " Oct 30 20:30:23.587885 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:23.587825 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f9c4f3a-c93c-4315-b0c0-95131a3413b6-home\" (UniqueName: \"kubernetes.io/empty-dir/8f9c4f3a-c93c-4315-b0c0-95131a3413b6-home\") pod \"8f9c4f3a-c93c-4315-b0c0-95131a3413b6\" (UID: \"8f9c4f3a-c93c-4315-b0c0-95131a3413b6\") " Oct 30 20:30:23.587944 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:23.587926 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f9c4f3a-c93c-4315-b0c0-95131a3413b6-tmp-dir" pod "8f9c4f3a-c93c-4315-b0c0-95131a3413b6" (UID: "8f9c4f3a-c93c-4315-b0c0-95131a3413b6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:30:23.588104 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:23.588059 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f9c4f3a-c93c-4315-b0c0-95131a3413b6-model-cache" pod "8f9c4f3a-c93c-4315-b0c0-95131a3413b6" (UID: "8f9c4f3a-c93c-4315-b0c0-95131a3413b6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:30:23.588212 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:23.588118 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f9c4f3a-c93c-4315-b0c0-95131a3413b6-home" pod "8f9c4f3a-c93c-4315-b0c0-95131a3413b6" (UID: "8f9c4f3a-c93c-4315-b0c0-95131a3413b6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:30:23.588327 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:23.588301 4292 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f9c4f3a-c93c-4315-b0c0-95131a3413b6-tmp-dir\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:30:23.588501 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:23.588337 4292 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f9c4f3a-c93c-4315-b0c0-95131a3413b6-model-cache\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:30:23.588501 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:23.588373 4292 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f9c4f3a-c93c-4315-b0c0-95131a3413b6-home\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:30:23.589903 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:23.589879 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f9c4f3a-c93c-4315-b0c0-95131a3413b6-dshm" pod "8f9c4f3a-c93c-4315-b0c0-95131a3413b6" (UID: "8f9c4f3a-c93c-4315-b0c0-95131a3413b6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:30:23.589994 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:23.589920 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f9c4f3a-c93c-4315-b0c0-95131a3413b6-tls-certs" pod "8f9c4f3a-c93c-4315-b0c0-95131a3413b6" (UID: "8f9c4f3a-c93c-4315-b0c0-95131a3413b6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:30:23.589994 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:23.589927 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f9c4f3a-c93c-4315-b0c0-95131a3413b6-kube-api-access-4ltwn" pod "8f9c4f3a-c93c-4315-b0c0-95131a3413b6" (UID: "8f9c4f3a-c93c-4315-b0c0-95131a3413b6"). InnerVolumeSpecName "kube-api-access-4ltwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:30:23.691392 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:23.691131 4292 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f9c4f3a-c93c-4315-b0c0-95131a3413b6-tls-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:30:23.691392 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:23.691172 4292 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f9c4f3a-c93c-4315-b0c0-95131a3413b6-dshm\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:30:23.691392 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:23.691188 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ltwn\" (UniqueName: \"kubernetes.io/projected/8f9c4f3a-c93c-4315-b0c0-95131a3413b6-kube-api-access-4ltwn\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:30:24.067048 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:24.067017 4292 generic.go:356] "Generic (PLEG): container finished" podID="8f9c4f3a-c93c-4315-b0c0-95131a3413b6" containerID="f2e8ca27df0a0687d642865898b7ee25281a80fae017bc184fc50577dd095cd9" exitCode=0 Oct 30 20:30:24.067233 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:24.067053 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-64120b/auth-test-model-64120b-kserve-8968677fc-q4rsd" event={"ID":"8f9c4f3a-c93c-4315-b0c0-95131a3413b6","Type":"ContainerDied","Data":"f2e8ca27df0a0687d642865898b7ee25281a80fae017bc184fc50577dd095cd9"} Oct 30 20:30:24.067233 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:24.067060 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-64120b/auth-test-model-64120b-kserve-8968677fc-q4rsd" Oct 30 20:30:24.067233 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:24.067080 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-64120b/auth-test-model-64120b-kserve-8968677fc-q4rsd" event={"ID":"8f9c4f3a-c93c-4315-b0c0-95131a3413b6","Type":"ContainerDied","Data":"4b00ab9b7da80444d09f39b560ad0a82661e3990584c5c68e41a941b6ac26ff3"} Oct 30 20:30:24.067233 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:24.067104 4292 scope.go:122] "RemoveContainer" containerID="f2e8ca27df0a0687d642865898b7ee25281a80fae017bc184fc50577dd095cd9" Oct 30 20:30:24.077317 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:24.077298 4292 scope.go:122] "RemoveContainer" containerID="f2e8ca27df0a0687d642865898b7ee25281a80fae017bc184fc50577dd095cd9" Oct 30 20:30:24.077615 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:30:24.077594 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2e8ca27df0a0687d642865898b7ee25281a80fae017bc184fc50577dd095cd9\": container with ID starting with f2e8ca27df0a0687d642865898b7ee25281a80fae017bc184fc50577dd095cd9 not found: ID does not exist" containerID="f2e8ca27df0a0687d642865898b7ee25281a80fae017bc184fc50577dd095cd9" Oct 30 20:30:24.077681 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:24.077627 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2e8ca27df0a0687d642865898b7ee25281a80fae017bc184fc50577dd095cd9"} err="failed to get container status \"f2e8ca27df0a0687d642865898b7ee25281a80fae017bc184fc50577dd095cd9\": rpc error: code = NotFound desc = could not find container \"f2e8ca27df0a0687d642865898b7ee25281a80fae017bc184fc50577dd095cd9\": container with ID starting with f2e8ca27df0a0687d642865898b7ee25281a80fae017bc184fc50577dd095cd9 not found: ID does not exist" Oct 30 20:30:24.236839 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:24.236801 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-64120b/auth-test-model-64120b-kserve-8968677fc-q4rsd"] Oct 30 20:30:24.287517 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:24.287483 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-64120b/auth-test-model-64120b-kserve-8968677fc-q4rsd"] Oct 30 20:30:24.582555 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:24.582521 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32b36613-26a9-4472-90fd-916bc95383b6" path="/var/lib/kubelet/pods/32b36613-26a9-4472-90fd-916bc95383b6/volumes" Oct 30 20:30:24.582971 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:24.582958 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f9c4f3a-c93c-4315-b0c0-95131a3413b6" path="/var/lib/kubelet/pods/8f9c4f3a-c93c-4315-b0c0-95131a3413b6/volumes" Oct 30 20:30:40.020403 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.020365 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b91df1-764b746c9f-hz7m2"] Oct 30 20:30:40.021057 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021037 4292 cpu_manager.go:414] "containerMap: removing container" podUID="32b36613-26a9-4472-90fd-916bc95383b6" containerName="cleanup" Oct 30 20:30:40.021151 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021059 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32b36613-26a9-4472-90fd-916bc95383b6" containerName="cleanup" Oct 30 20:30:40.021151 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021078 4292 cpu_manager.go:414] "containerMap: removing container" podUID="062d2841-086a-493c-8bd4-e26a0f3c82ff" containerName="cleanup" Oct 30 20:30:40.021151 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021086 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="062d2841-086a-493c-8bd4-e26a0f3c82ff" containerName="cleanup" Oct 30 20:30:40.021151 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021101 4292 cpu_manager.go:414] "containerMap: removing container" podUID="9d3d5c8c-e915-424c-a674-07c2bfb01a7d" containerName="cleanup" Oct 30 20:30:40.021151 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021109 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d3d5c8c-e915-424c-a674-07c2bfb01a7d" containerName="cleanup" Oct 30 20:30:40.021151 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021124 4292 cpu_manager.go:414] "containerMap: removing container" podUID="9d3d5c8c-e915-424c-a674-07c2bfb01a7d" containerName="cleanup" Oct 30 20:30:40.021151 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021131 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d3d5c8c-e915-424c-a674-07c2bfb01a7d" containerName="cleanup" Oct 30 20:30:40.021151 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021142 4292 cpu_manager.go:414] "containerMap: removing container" podUID="57e5a5eb-f9ae-42f6-8c6e-e4336b81c953" containerName="cleanup" Oct 30 20:30:40.021151 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021149 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57e5a5eb-f9ae-42f6-8c6e-e4336b81c953" containerName="cleanup" Oct 30 20:30:40.021699 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021158 4292 cpu_manager.go:414] "containerMap: removing container" podUID="79ffe45a-2987-49cc-909c-920ccbbbb53a" containerName="cleanup" Oct 30 20:30:40.021699 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021165 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79ffe45a-2987-49cc-909c-920ccbbbb53a" containerName="cleanup" Oct 30 20:30:40.021699 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021177 4292 cpu_manager.go:414] "containerMap: removing container" podUID="79ffe45a-2987-49cc-909c-920ccbbbb53a" containerName="cleanup" Oct 30 20:30:40.021699 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021184 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79ffe45a-2987-49cc-909c-920ccbbbb53a" containerName="cleanup" Oct 30 20:30:40.021699 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021193 4292 cpu_manager.go:414] "containerMap: removing container" podUID="ff6b576c-bc3c-4a15-ae30-57474f6285d0" containerName="cleanup" Oct 30 20:30:40.021699 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021200 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff6b576c-bc3c-4a15-ae30-57474f6285d0" containerName="cleanup" Oct 30 20:30:40.021699 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021211 4292 cpu_manager.go:414] "containerMap: removing container" podUID="a0b2551f-95d5-4a8a-b6e5-08cd9a4d9572" containerName="cleanup" Oct 30 20:30:40.021699 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021218 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0b2551f-95d5-4a8a-b6e5-08cd9a4d9572" containerName="cleanup" Oct 30 20:30:40.021699 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021230 4292 cpu_manager.go:414] "containerMap: removing container" podUID="ff6b576c-bc3c-4a15-ae30-57474f6285d0" containerName="cleanup" Oct 30 20:30:40.021699 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021238 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff6b576c-bc3c-4a15-ae30-57474f6285d0" containerName="cleanup" Oct 30 20:30:40.021699 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021252 4292 cpu_manager.go:414] "containerMap: removing container" podUID="8f9c4f3a-c93c-4315-b0c0-95131a3413b6" containerName="main" Oct 30 20:30:40.021699 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021260 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f9c4f3a-c93c-4315-b0c0-95131a3413b6" containerName="main" Oct 30 20:30:40.021699 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021271 4292 cpu_manager.go:414] "containerMap: removing container" podUID="57e5a5eb-f9ae-42f6-8c6e-e4336b81c953" containerName="cleanup" Oct 30 20:30:40.021699 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021278 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57e5a5eb-f9ae-42f6-8c6e-e4336b81c953" containerName="cleanup" Oct 30 20:30:40.021699 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021294 4292 cpu_manager.go:414] "containerMap: removing container" podUID="062d2841-086a-493c-8bd4-e26a0f3c82ff" containerName="cleanup" Oct 30 20:30:40.021699 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021302 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="062d2841-086a-493c-8bd4-e26a0f3c82ff" containerName="cleanup" Oct 30 20:30:40.021699 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021311 4292 cpu_manager.go:414] "containerMap: removing container" podUID="32b36613-26a9-4472-90fd-916bc95383b6" containerName="cleanup" Oct 30 20:30:40.021699 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021317 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32b36613-26a9-4472-90fd-916bc95383b6" containerName="cleanup" Oct 30 20:30:40.021699 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021326 4292 cpu_manager.go:414] "containerMap: removing container" podUID="a895dc65-411f-468e-8872-3fb6b6e35a5e" containerName="cleanup" Oct 30 20:30:40.021699 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021341 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a895dc65-411f-468e-8872-3fb6b6e35a5e" containerName="cleanup" Oct 30 20:30:40.021699 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021382 4292 cpu_manager.go:414] "containerMap: removing container" podUID="062d2841-086a-493c-8bd4-e26a0f3c82ff" containerName="cleanup" Oct 30 20:30:40.021699 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021389 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="062d2841-086a-493c-8bd4-e26a0f3c82ff" containerName="cleanup" Oct 30 20:30:40.021699 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021401 4292 cpu_manager.go:414] "containerMap: removing container" podUID="79ffe45a-2987-49cc-909c-920ccbbbb53a" containerName="cleanup" Oct 30 20:30:40.021699 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021408 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79ffe45a-2987-49cc-909c-920ccbbbb53a" containerName="cleanup" Oct 30 20:30:40.021699 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021418 4292 cpu_manager.go:414] "containerMap: removing container" podUID="bc4d4a92-2ef3-42a5-82be-70bb892011b1" containerName="cleanup" Oct 30 20:30:40.021699 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021426 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc4d4a92-2ef3-42a5-82be-70bb892011b1" containerName="cleanup" Oct 30 20:30:40.021699 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021439 4292 cpu_manager.go:414] "containerMap: removing container" podUID="a895dc65-411f-468e-8872-3fb6b6e35a5e" containerName="cleanup" Oct 30 20:30:40.021699 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021446 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a895dc65-411f-468e-8872-3fb6b6e35a5e" containerName="cleanup" Oct 30 20:30:40.021699 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021459 4292 cpu_manager.go:414] "containerMap: removing container" podUID="bc4d4a92-2ef3-42a5-82be-70bb892011b1" containerName="cleanup" Oct 30 20:30:40.021699 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021467 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc4d4a92-2ef3-42a5-82be-70bb892011b1" containerName="cleanup" Oct 30 20:30:40.021699 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021476 4292 cpu_manager.go:414] "containerMap: removing container" podUID="32b36613-26a9-4472-90fd-916bc95383b6" containerName="cleanup" Oct 30 20:30:40.021699 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021483 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32b36613-26a9-4472-90fd-916bc95383b6" containerName="cleanup" Oct 30 20:30:40.021699 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021499 4292 cpu_manager.go:414] "containerMap: removing container" podUID="bc4d4a92-2ef3-42a5-82be-70bb892011b1" containerName="cleanup" Oct 30 20:30:40.021699 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021506 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc4d4a92-2ef3-42a5-82be-70bb892011b1" containerName="cleanup" Oct 30 20:30:40.021699 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021524 4292 cpu_manager.go:414] "containerMap: removing container" podUID="a0b2551f-95d5-4a8a-b6e5-08cd9a4d9572" containerName="cleanup" Oct 30 20:30:40.021699 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021532 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0b2551f-95d5-4a8a-b6e5-08cd9a4d9572" containerName="cleanup" Oct 30 20:30:40.021699 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021542 4292 cpu_manager.go:414] "containerMap: removing container" podUID="a895dc65-411f-468e-8872-3fb6b6e35a5e" containerName="cleanup" Oct 30 20:30:40.021699 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021549 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a895dc65-411f-468e-8872-3fb6b6e35a5e" containerName="cleanup" Oct 30 20:30:40.021699 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021564 4292 cpu_manager.go:414] "containerMap: removing container" podUID="a0b2551f-95d5-4a8a-b6e5-08cd9a4d9572" containerName="cleanup" Oct 30 20:30:40.021699 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021571 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0b2551f-95d5-4a8a-b6e5-08cd9a4d9572" containerName="cleanup" Oct 30 20:30:40.021699 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021705 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0b2551f-95d5-4a8a-b6e5-08cd9a4d9572" containerName="cleanup" Oct 30 20:30:40.021699 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021720 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="32b36613-26a9-4472-90fd-916bc95383b6" containerName="cleanup" Oct 30 20:30:40.023666 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021730 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="062d2841-086a-493c-8bd4-e26a0f3c82ff" containerName="cleanup" Oct 30 20:30:40.023666 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021741 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d3d5c8c-e915-424c-a674-07c2bfb01a7d" containerName="cleanup" Oct 30 20:30:40.023666 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021751 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d3d5c8c-e915-424c-a674-07c2bfb01a7d" containerName="cleanup" Oct 30 20:30:40.023666 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021760 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="57e5a5eb-f9ae-42f6-8c6e-e4336b81c953" containerName="cleanup" Oct 30 20:30:40.023666 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021768 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="79ffe45a-2987-49cc-909c-920ccbbbb53a" containerName="cleanup" Oct 30 20:30:40.023666 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021776 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="79ffe45a-2987-49cc-909c-920ccbbbb53a" containerName="cleanup" Oct 30 20:30:40.023666 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021784 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff6b576c-bc3c-4a15-ae30-57474f6285d0" containerName="cleanup" Oct 30 20:30:40.023666 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021792 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff6b576c-bc3c-4a15-ae30-57474f6285d0" containerName="cleanup" Oct 30 20:30:40.023666 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021803 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f9c4f3a-c93c-4315-b0c0-95131a3413b6" containerName="main" Oct 30 20:30:40.023666 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021813 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="32b36613-26a9-4472-90fd-916bc95383b6" containerName="cleanup" Oct 30 20:30:40.023666 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021821 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="a895dc65-411f-468e-8872-3fb6b6e35a5e" containerName="cleanup" Oct 30 20:30:40.023666 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021829 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="062d2841-086a-493c-8bd4-e26a0f3c82ff" containerName="cleanup" Oct 30 20:30:40.023666 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021837 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc4d4a92-2ef3-42a5-82be-70bb892011b1" containerName="cleanup" Oct 30 20:30:40.023666 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021845 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="a895dc65-411f-468e-8872-3fb6b6e35a5e" containerName="cleanup" Oct 30 20:30:40.023666 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021853 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc4d4a92-2ef3-42a5-82be-70bb892011b1" containerName="cleanup" Oct 30 20:30:40.023666 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021863 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc4d4a92-2ef3-42a5-82be-70bb892011b1" containerName="cleanup" Oct 30 20:30:40.023666 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021872 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0b2551f-95d5-4a8a-b6e5-08cd9a4d9572" containerName="cleanup" Oct 30 20:30:40.023666 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.021881 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="57e5a5eb-f9ae-42f6-8c6e-e4336b81c953" containerName="cleanup" Oct 30 20:30:40.025775 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.025754 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b91df1-764b746c9f-hz7m2" Oct 30 20:30:40.078142 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.078108 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/527ba566-957f-406d-8756-ce938d0c3922-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-b91df1-764b746c9f-hz7m2\" (UID: \"527ba566-957f-406d-8756-ce938d0c3922\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b91df1-764b746c9f-hz7m2" Oct 30 20:30:40.078414 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.078286 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/527ba566-957f-406d-8756-ce938d0c3922-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-b91df1-764b746c9f-hz7m2\" (UID: \"527ba566-957f-406d-8756-ce938d0c3922\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b91df1-764b746c9f-hz7m2" Oct 30 20:30:40.078414 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.078322 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/527ba566-957f-406d-8756-ce938d0c3922-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-b91df1-764b746c9f-hz7m2\" (UID: \"527ba566-957f-406d-8756-ce938d0c3922\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b91df1-764b746c9f-hz7m2" Oct 30 20:30:40.078414 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.078378 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv6b4\" (UniqueName: \"kubernetes.io/projected/527ba566-957f-406d-8756-ce938d0c3922-kube-api-access-hv6b4\") pod \"maas-api-e2e-shared-b-w4-b91df1-764b746c9f-hz7m2\" (UID: \"527ba566-957f-406d-8756-ce938d0c3922\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b91df1-764b746c9f-hz7m2" Oct 30 20:30:40.080042 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.080013 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b91df1-764b746c9f-hz7m2"] Oct 30 20:30:40.114667 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.114642 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-b91df1-serving-cert\"" Oct 30 20:30:40.179589 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.179551 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/527ba566-957f-406d-8756-ce938d0c3922-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-b91df1-764b746c9f-hz7m2\" (UID: \"527ba566-957f-406d-8756-ce938d0c3922\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b91df1-764b746c9f-hz7m2" Oct 30 20:30:40.179778 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.179662 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/527ba566-957f-406d-8756-ce938d0c3922-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-b91df1-764b746c9f-hz7m2\" (UID: \"527ba566-957f-406d-8756-ce938d0c3922\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b91df1-764b746c9f-hz7m2" Oct 30 20:30:40.179778 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.179694 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/527ba566-957f-406d-8756-ce938d0c3922-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-b91df1-764b746c9f-hz7m2\" (UID: \"527ba566-957f-406d-8756-ce938d0c3922\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b91df1-764b746c9f-hz7m2" Oct 30 20:30:40.179778 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.179721 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv6b4\" (UniqueName: \"kubernetes.io/projected/527ba566-957f-406d-8756-ce938d0c3922-kube-api-access-hv6b4\") pod \"maas-api-e2e-shared-b-w4-b91df1-764b746c9f-hz7m2\" (UID: \"527ba566-957f-406d-8756-ce938d0c3922\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b91df1-764b746c9f-hz7m2" Oct 30 20:30:40.179778 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:30:40.179774 4292 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b91df1-serving-cert: secret "maas-api-e2e-shared-b-w4-b91df1-serving-cert" not found Oct 30 20:30:40.180011 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:30:40.179842 4292 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/527ba566-957f-406d-8756-ce938d0c3922-maas-api-tls podName:527ba566-957f-406d-8756-ce938d0c3922 nodeName:}" failed. No retries permitted until 2026-10-30 20:30:40.679825424 +0000 UTC m=+3944.198140864 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/527ba566-957f-406d-8756-ce938d0c3922-maas-api-tls") pod "maas-api-e2e-shared-b-w4-b91df1-764b746c9f-hz7m2" (UID: "527ba566-957f-406d-8756-ce938d0c3922") : secret "maas-api-e2e-shared-b-w4-b91df1-serving-cert" not found Oct 30 20:30:40.180555 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.180531 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/527ba566-957f-406d-8756-ce938d0c3922-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-b91df1-764b746c9f-hz7m2\" (UID: \"527ba566-957f-406d-8756-ce938d0c3922\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b91df1-764b746c9f-hz7m2" Oct 30 20:30:40.182041 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.182016 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/527ba566-957f-406d-8756-ce938d0c3922-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-b91df1-764b746c9f-hz7m2\" (UID: \"527ba566-957f-406d-8756-ce938d0c3922\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b91df1-764b746c9f-hz7m2" Oct 30 20:30:40.242767 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.242746 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv6b4\" (UniqueName: \"kubernetes.io/projected/527ba566-957f-406d-8756-ce938d0c3922-kube-api-access-hv6b4\") pod \"maas-api-e2e-shared-b-w4-b91df1-764b746c9f-hz7m2\" (UID: \"527ba566-957f-406d-8756-ce938d0c3922\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b91df1-764b746c9f-hz7m2" Oct 30 20:30:40.689535 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.689503 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/527ba566-957f-406d-8756-ce938d0c3922-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-b91df1-764b746c9f-hz7m2\" (UID: \"527ba566-957f-406d-8756-ce938d0c3922\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b91df1-764b746c9f-hz7m2" Oct 30 20:30:40.691958 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.691927 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/527ba566-957f-406d-8756-ce938d0c3922-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-b91df1-764b746c9f-hz7m2\" (UID: \"527ba566-957f-406d-8756-ce938d0c3922\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b91df1-764b746c9f-hz7m2" Oct 30 20:30:40.938392 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:40.938338 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b91df1-764b746c9f-hz7m2" Oct 30 20:30:41.309412 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:30:41.309304 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod527ba566_957f_406d_8756_ce938d0c3922.slice/crio-8d37d153d369bb51203e2982bfc45c452c7bcee8639b66bed08d93d29aedd2bb WatchSource:0}: Error finding container 8d37d153d369bb51203e2982bfc45c452c7bcee8639b66bed08d93d29aedd2bb: Status 404 returned error can't find the container with id 8d37d153d369bb51203e2982bfc45c452c7bcee8639b66bed08d93d29aedd2bb Oct 30 20:30:41.310897 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:41.310873 4292 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 20:30:41.311920 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:41.311839 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b91df1-764b746c9f-hz7m2"] Oct 30 20:30:42.179136 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:42.179105 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b91df1-764b746c9f-hz7m2" event={"ID":"527ba566-957f-406d-8756-ce938d0c3922","Type":"ContainerStarted","Data":"8d37d153d369bb51203e2982bfc45c452c7bcee8639b66bed08d93d29aedd2bb"} Oct 30 20:30:43.198371 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:43.198131 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b91df1-764b746c9f-hz7m2" event={"ID":"527ba566-957f-406d-8756-ce938d0c3922","Type":"ContainerStarted","Data":"4d7d9eaef73b4abb2270be44990a74ec18acdee9692f41ea49b3897d603ebc94"} Oct 30 20:30:43.203370 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:43.200457 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b91df1-764b746c9f-hz7m2" Oct 30 20:30:43.240210 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:30:43.240174 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:30:43.321456 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:43.321407 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b91df1-764b746c9f-hz7m2" podStartSLOduration=3.5106155340000003 podStartE2EDuration="4.321376506s" totalImagesPullingTime="810.760972ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:30:43.320649238 +0000 UTC m=+3946.838964693" watchObservedRunningTime="2026-10-30 20:30:43.321376506 +0000 UTC m=+3946.839691960" Oct 30 20:30:50.215064 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:30:50.215031 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b91df1-764b746c9f-hz7m2" Oct 30 20:31:05.243235 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:05.243194 4292 patch_prober.go:29] interesting pod/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:31:05.243661 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:05.243252 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d" podUID="476ee650-d28b-4580-9eb4-a9f095df7822" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:31:06.333566 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:06.333535 4292 generic.go:356] "Generic (PLEG): container finished" podID="476ee650-d28b-4580-9eb4-a9f095df7822" containerID="dd97bc70b890574bee4c8fbf27e6c305188815e9e0c39c0e353c371201c010bc" exitCode=0 Oct 30 20:31:06.334103 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:06.333609 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d" event={"ID":"476ee650-d28b-4580-9eb4-a9f095df7822","Type":"ContainerDied","Data":"dd97bc70b890574bee4c8fbf27e6c305188815e9e0c39c0e353c371201c010bc"} Oct 30 20:31:06.334176 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:06.334111 4292 scope.go:122] "RemoveContainer" containerID="dd97bc70b890574bee4c8fbf27e6c305188815e9e0c39c0e353c371201c010bc" Oct 30 20:31:06.335800 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:06.335776 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:31:06.335872 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:06.335848 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:31:06.335907 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:06.335882 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:31:07.341749 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:07.341713 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d" event={"ID":"476ee650-d28b-4580-9eb4-a9f095df7822","Type":"ContainerStarted","Data":"45750ddbb4237d69d75e88edb1ec8d5d39c3fc2f221cdc744cabf0ad758416f6"} Oct 30 20:31:07.707565 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:07.707489 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d" Oct 30 20:31:07.707714 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:07.707642 4292 patch_prober.go:29] interesting pod/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Oct 30 20:31:07.707714 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:07.707690 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d" podUID="476ee650-d28b-4580-9eb4-a9f095df7822" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Oct 30 20:31:08.707959 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:08.707928 4292 patch_prober.go:29] interesting pod/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Oct 30 20:31:08.708458 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:08.707981 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d" podUID="476ee650-d28b-4580-9eb4-a9f095df7822" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Oct 30 20:31:09.707904 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:09.707867 4292 patch_prober.go:29] interesting pod/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Oct 30 20:31:09.708196 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:09.708169 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d" podUID="476ee650-d28b-4580-9eb4-a9f095df7822" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Oct 30 20:31:09.861059 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:09.861006 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms"] Oct 30 20:31:09.864014 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:09.862065 4292 cpu_manager.go:414] "containerMap: removing container" podUID="ff6b576c-bc3c-4a15-ae30-57474f6285d0" containerName="cleanup" Oct 30 20:31:09.864014 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:09.862093 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff6b576c-bc3c-4a15-ae30-57474f6285d0" containerName="cleanup" Oct 30 20:31:09.864014 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:09.862116 4292 cpu_manager.go:414] "containerMap: removing container" podUID="57e5a5eb-f9ae-42f6-8c6e-e4336b81c953" containerName="cleanup" Oct 30 20:31:09.864014 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:09.862126 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57e5a5eb-f9ae-42f6-8c6e-e4336b81c953" containerName="cleanup" Oct 30 20:31:09.864014 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:09.862190 4292 cpu_manager.go:414] "containerMap: removing container" podUID="9d3d5c8c-e915-424c-a674-07c2bfb01a7d" containerName="cleanup" Oct 30 20:31:09.864014 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:09.862199 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d3d5c8c-e915-424c-a674-07c2bfb01a7d" containerName="cleanup" Oct 30 20:31:09.864014 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:09.862312 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="32b36613-26a9-4472-90fd-916bc95383b6" containerName="cleanup" Oct 30 20:31:09.864014 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:09.862327 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="062d2841-086a-493c-8bd4-e26a0f3c82ff" containerName="cleanup" Oct 30 20:31:09.864014 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:09.862337 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="a895dc65-411f-468e-8872-3fb6b6e35a5e" containerName="cleanup" Oct 30 20:31:09.864014 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:09.862369 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0b2551f-95d5-4a8a-b6e5-08cd9a4d9572" containerName="cleanup" Oct 30 20:31:09.864014 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:09.862386 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d3d5c8c-e915-424c-a674-07c2bfb01a7d" containerName="cleanup" Oct 30 20:31:09.864014 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:09.862398 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="57e5a5eb-f9ae-42f6-8c6e-e4336b81c953" containerName="cleanup" Oct 30 20:31:09.864014 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:09.862406 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="79ffe45a-2987-49cc-909c-920ccbbbb53a" containerName="cleanup" Oct 30 20:31:09.864014 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:09.862416 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff6b576c-bc3c-4a15-ae30-57474f6285d0" containerName="cleanup" Oct 30 20:31:09.871429 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:09.870445 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms" Oct 30 20:31:09.914776 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:09.914736 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms"] Oct 30 20:31:09.969699 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:09.969601 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-e391de-openshift-default-dockercfg-2pqng\"" Oct 30 20:31:10.020057 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:10.019411 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d5226bf-9ab0-496b-b2f5-f0d730554365-istio-envoy\") pod \"e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms\" (UID: \"9d5226bf-9ab0-496b-b2f5-f0d730554365\") " pod="openshift-ingress/e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms" Oct 30 20:31:10.020057 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:10.019471 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d5226bf-9ab0-496b-b2f5-f0d730554365-istiod-ca-cert\") pod \"e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms\" (UID: \"9d5226bf-9ab0-496b-b2f5-f0d730554365\") " pod="openshift-ingress/e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms" Oct 30 20:31:10.020057 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:10.019518 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d5226bf-9ab0-496b-b2f5-f0d730554365-workload-socket\") pod \"e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms\" (UID: \"9d5226bf-9ab0-496b-b2f5-f0d730554365\") " pod="openshift-ingress/e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms" Oct 30 20:31:10.020057 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:10.019548 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7hhw\" (UniqueName: \"kubernetes.io/projected/9d5226bf-9ab0-496b-b2f5-f0d730554365-kube-api-access-t7hhw\") pod \"e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms\" (UID: \"9d5226bf-9ab0-496b-b2f5-f0d730554365\") " pod="openshift-ingress/e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms" Oct 30 20:31:10.020057 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:10.019587 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d5226bf-9ab0-496b-b2f5-f0d730554365-workload-certs\") pod \"e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms\" (UID: \"9d5226bf-9ab0-496b-b2f5-f0d730554365\") " pod="openshift-ingress/e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms" Oct 30 20:31:10.020057 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:10.019674 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d5226bf-9ab0-496b-b2f5-f0d730554365-istio-token\") pod \"e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms\" (UID: \"9d5226bf-9ab0-496b-b2f5-f0d730554365\") " pod="openshift-ingress/e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms" Oct 30 20:31:10.020057 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:10.019751 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d5226bf-9ab0-496b-b2f5-f0d730554365-credential-socket\") pod \"e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms\" (UID: \"9d5226bf-9ab0-496b-b2f5-f0d730554365\") " pod="openshift-ingress/e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms" Oct 30 20:31:10.020057 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:10.019798 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d5226bf-9ab0-496b-b2f5-f0d730554365-istio-data\") pod \"e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms\" (UID: \"9d5226bf-9ab0-496b-b2f5-f0d730554365\") " pod="openshift-ingress/e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms" Oct 30 20:31:10.020057 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:10.019876 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d5226bf-9ab0-496b-b2f5-f0d730554365-istio-podinfo\") pod \"e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms\" (UID: \"9d5226bf-9ab0-496b-b2f5-f0d730554365\") " pod="openshift-ingress/e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms" Oct 30 20:31:10.121102 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:10.121063 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d5226bf-9ab0-496b-b2f5-f0d730554365-istio-token\") pod \"e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms\" (UID: \"9d5226bf-9ab0-496b-b2f5-f0d730554365\") " pod="openshift-ingress/e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms" Oct 30 20:31:10.121392 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:10.121143 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d5226bf-9ab0-496b-b2f5-f0d730554365-credential-socket\") pod \"e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms\" (UID: \"9d5226bf-9ab0-496b-b2f5-f0d730554365\") " pod="openshift-ingress/e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms" Oct 30 20:31:10.121392 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:10.121183 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d5226bf-9ab0-496b-b2f5-f0d730554365-istio-data\") pod \"e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms\" (UID: \"9d5226bf-9ab0-496b-b2f5-f0d730554365\") " pod="openshift-ingress/e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms" Oct 30 20:31:10.121392 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:10.121243 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d5226bf-9ab0-496b-b2f5-f0d730554365-istio-podinfo\") pod \"e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms\" (UID: \"9d5226bf-9ab0-496b-b2f5-f0d730554365\") " pod="openshift-ingress/e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms" Oct 30 20:31:10.121392 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:10.121272 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d5226bf-9ab0-496b-b2f5-f0d730554365-istio-envoy\") pod \"e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms\" (UID: \"9d5226bf-9ab0-496b-b2f5-f0d730554365\") " pod="openshift-ingress/e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms" Oct 30 20:31:10.121392 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:10.121303 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d5226bf-9ab0-496b-b2f5-f0d730554365-istiod-ca-cert\") pod \"e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms\" (UID: \"9d5226bf-9ab0-496b-b2f5-f0d730554365\") " pod="openshift-ingress/e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms" Oct 30 20:31:10.121392 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:10.121379 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d5226bf-9ab0-496b-b2f5-f0d730554365-workload-socket\") pod \"e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms\" (UID: \"9d5226bf-9ab0-496b-b2f5-f0d730554365\") " pod="openshift-ingress/e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms" Oct 30 20:31:10.121724 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:10.121409 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7hhw\" (UniqueName: \"kubernetes.io/projected/9d5226bf-9ab0-496b-b2f5-f0d730554365-kube-api-access-t7hhw\") pod \"e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms\" (UID: \"9d5226bf-9ab0-496b-b2f5-f0d730554365\") " pod="openshift-ingress/e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms" Oct 30 20:31:10.121724 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:10.121441 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d5226bf-9ab0-496b-b2f5-f0d730554365-workload-certs\") pod \"e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms\" (UID: \"9d5226bf-9ab0-496b-b2f5-f0d730554365\") " pod="openshift-ingress/e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms" Oct 30 20:31:10.121952 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:10.121926 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d5226bf-9ab0-496b-b2f5-f0d730554365-workload-certs\") pod \"e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms\" (UID: \"9d5226bf-9ab0-496b-b2f5-f0d730554365\") " pod="openshift-ingress/e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms" Oct 30 20:31:10.122334 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:10.122280 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d5226bf-9ab0-496b-b2f5-f0d730554365-credential-socket\") pod \"e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms\" (UID: \"9d5226bf-9ab0-496b-b2f5-f0d730554365\") " pod="openshift-ingress/e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms" Oct 30 20:31:10.122479 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:10.122330 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d5226bf-9ab0-496b-b2f5-f0d730554365-workload-socket\") pod \"e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms\" (UID: \"9d5226bf-9ab0-496b-b2f5-f0d730554365\") " pod="openshift-ingress/e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms" Oct 30 20:31:10.122728 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:10.122702 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d5226bf-9ab0-496b-b2f5-f0d730554365-istio-data\") pod \"e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms\" (UID: \"9d5226bf-9ab0-496b-b2f5-f0d730554365\") " pod="openshift-ingress/e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms" Oct 30 20:31:10.123202 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:10.123181 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d5226bf-9ab0-496b-b2f5-f0d730554365-istiod-ca-cert\") pod \"e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms\" (UID: \"9d5226bf-9ab0-496b-b2f5-f0d730554365\") " pod="openshift-ingress/e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms" Oct 30 20:31:10.125395 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:10.125139 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d5226bf-9ab0-496b-b2f5-f0d730554365-istio-podinfo\") pod \"e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms\" (UID: \"9d5226bf-9ab0-496b-b2f5-f0d730554365\") " pod="openshift-ingress/e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms" Oct 30 20:31:10.125395 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:10.125381 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d5226bf-9ab0-496b-b2f5-f0d730554365-istio-envoy\") pod \"e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms\" (UID: \"9d5226bf-9ab0-496b-b2f5-f0d730554365\") " pod="openshift-ingress/e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms" Oct 30 20:31:10.184798 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:10.184765 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d5226bf-9ab0-496b-b2f5-f0d730554365-istio-token\") pod \"e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms\" (UID: \"9d5226bf-9ab0-496b-b2f5-f0d730554365\") " pod="openshift-ingress/e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms" Oct 30 20:31:10.184993 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:10.184925 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7hhw\" (UniqueName: \"kubernetes.io/projected/9d5226bf-9ab0-496b-b2f5-f0d730554365-kube-api-access-t7hhw\") pod \"e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms\" (UID: \"9d5226bf-9ab0-496b-b2f5-f0d730554365\") " pod="openshift-ingress/e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms" Oct 30 20:31:10.212099 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:10.212069 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms" Oct 30 20:31:10.664376 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:10.663757 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms"] Oct 30 20:31:10.676371 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:31:10.676006 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d5226bf_9ab0_496b_b2f5_f0d730554365.slice/crio-d7193d5200163cc2379cd35a216422439f048745a331db8326dc9aab1c1a1dc4 WatchSource:0}: Error finding container d7193d5200163cc2379cd35a216422439f048745a331db8326dc9aab1c1a1dc4: Status 404 returned error can't find the container with id d7193d5200163cc2379cd35a216422439f048745a331db8326dc9aab1c1a1dc4 Oct 30 20:31:10.684279 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:10.684235 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:31:10.684436 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:10.684332 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:31:10.684436 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:10.684398 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:31:10.707668 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:10.707643 4292 patch_prober.go:29] interesting pod/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Oct 30 20:31:10.707773 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:10.707694 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d" podUID="476ee650-d28b-4580-9eb4-a9f095df7822" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Oct 30 20:31:11.394551 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:11.394514 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms" event={"ID":"9d5226bf-9ab0-496b-b2f5-f0d730554365","Type":"ContainerStarted","Data":"ac68c6895d5768fa483ea1155501e73eb48cea63371569daa1ec1260f43328ba"} Oct 30 20:31:11.394986 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:11.394576 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms" event={"ID":"9d5226bf-9ab0-496b-b2f5-f0d730554365","Type":"ContainerStarted","Data":"d7193d5200163cc2379cd35a216422439f048745a331db8326dc9aab1c1a1dc4"} Oct 30 20:31:11.519112 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:11.519059 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms" podStartSLOduration=2.5190449839999998 podStartE2EDuration="2.519044984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:31:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:31:11.516750571 +0000 UTC m=+3975.035066067" watchObservedRunningTime="2026-10-30 20:31:11.519044984 +0000 UTC m=+3975.037360440" Oct 30 20:31:11.708266 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:11.708184 4292 patch_prober.go:29] interesting pod/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Oct 30 20:31:11.708266 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:11.708242 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d" podUID="476ee650-d28b-4580-9eb4-a9f095df7822" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Oct 30 20:31:12.213065 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:12.213030 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms" Oct 30 20:31:12.219370 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:12.219327 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms" Oct 30 20:31:12.404756 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:12.404725 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms" Oct 30 20:31:12.405794 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:12.405773 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms" Oct 30 20:31:12.708093 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:12.708058 4292 patch_prober.go:29] interesting pod/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Oct 30 20:31:12.708249 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:12.708109 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d" podUID="476ee650-d28b-4580-9eb4-a9f095df7822" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Oct 30 20:31:13.220680 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:31:13.220645 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:31:13.712329 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:13.712295 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d" Oct 30 20:31:13.712847 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:13.712708 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d" Oct 30 20:31:13.713270 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:13.713252 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d" Oct 30 20:31:43.221182 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:31:43.221096 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:31:54.889958 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:54.889868 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9389b9-jmjk8"] Oct 30 20:31:54.893965 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:54.893943 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9389b9-jmjk8" Oct 30 20:31:54.948823 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:54.948799 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:31:54.993443 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:54.993401 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36fefbfb-420f-48d3-ade3-dede3da5b8e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-9389b9-jmjk8\" (UID: \"36fefbfb-420f-48d3-ade3-dede3da5b8e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9389b9-jmjk8" Oct 30 20:31:54.996821 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:54.996790 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcgrm\" (UniqueName: \"kubernetes.io/projected/36fefbfb-420f-48d3-ade3-dede3da5b8e7-kube-api-access-pcgrm\") pod \"maas-api-revoke-keys-e2e-extmod-a-9389b9-jmjk8\" (UID: \"36fefbfb-420f-48d3-ade3-dede3da5b8e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9389b9-jmjk8" Oct 30 20:31:55.002666 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:54.997241 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36fefbfb-420f-48d3-ade3-dede3da5b8e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-9389b9-jmjk8\" (UID: \"36fefbfb-420f-48d3-ade3-dede3da5b8e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9389b9-jmjk8" Oct 30 20:31:55.018105 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:55.018072 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\"" Oct 30 20:31:55.030024 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:55.029989 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9389b9-jmjk8"] Oct 30 20:31:55.119102 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:55.119071 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36fefbfb-420f-48d3-ade3-dede3da5b8e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-9389b9-jmjk8\" (UID: \"36fefbfb-420f-48d3-ade3-dede3da5b8e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9389b9-jmjk8" Oct 30 20:31:55.119276 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:55.119140 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36fefbfb-420f-48d3-ade3-dede3da5b8e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-9389b9-jmjk8\" (UID: \"36fefbfb-420f-48d3-ade3-dede3da5b8e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9389b9-jmjk8" Oct 30 20:31:55.119276 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:55.119168 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcgrm\" (UniqueName: \"kubernetes.io/projected/36fefbfb-420f-48d3-ade3-dede3da5b8e7-kube-api-access-pcgrm\") pod \"maas-api-revoke-keys-e2e-extmod-a-9389b9-jmjk8\" (UID: \"36fefbfb-420f-48d3-ade3-dede3da5b8e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9389b9-jmjk8" Oct 30 20:31:55.119901 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:55.119865 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36fefbfb-420f-48d3-ade3-dede3da5b8e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-9389b9-jmjk8\" (UID: \"36fefbfb-420f-48d3-ade3-dede3da5b8e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9389b9-jmjk8" Oct 30 20:31:55.121597 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:55.121580 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36fefbfb-420f-48d3-ade3-dede3da5b8e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-9389b9-jmjk8\" (UID: \"36fefbfb-420f-48d3-ade3-dede3da5b8e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9389b9-jmjk8" Oct 30 20:31:55.181652 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:55.181587 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcgrm\" (UniqueName: \"kubernetes.io/projected/36fefbfb-420f-48d3-ade3-dede3da5b8e7-kube-api-access-pcgrm\") pod \"maas-api-revoke-keys-e2e-extmod-a-9389b9-jmjk8\" (UID: \"36fefbfb-420f-48d3-ade3-dede3da5b8e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9389b9-jmjk8" Oct 30 20:31:55.207500 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:55.207472 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9389b9-jmjk8" Oct 30 20:31:55.577669 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:31:55.577633 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36fefbfb_420f_48d3_ade3_dede3da5b8e7.slice/crio-570c8a868d88df2a76938b046491e9b9e5e2771ac4247180b1340d1e3ee54a98 WatchSource:0}: Error finding container 570c8a868d88df2a76938b046491e9b9e5e2771ac4247180b1340d1e3ee54a98: Status 404 returned error can't find the container with id 570c8a868d88df2a76938b046491e9b9e5e2771ac4247180b1340d1e3ee54a98 Oct 30 20:31:55.580832 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:55.580813 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9389b9-jmjk8"] Oct 30 20:31:55.756573 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:55.756538 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9389b9-jmjk8" event={"ID":"36fefbfb-420f-48d3-ade3-dede3da5b8e7","Type":"ContainerStarted","Data":"570c8a868d88df2a76938b046491e9b9e5e2771ac4247180b1340d1e3ee54a98"} Oct 30 20:31:56.767851 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:56.766469 4292 generic.go:356] "Generic (PLEG): container finished" podID="36fefbfb-420f-48d3-ade3-dede3da5b8e7" containerID="b0c5e9641c82a20cdbdcb390e3be090df0efece3230889829254871c960dcefe" exitCode=0 Oct 30 20:31:56.767851 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:56.766633 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9389b9-jmjk8" event={"ID":"36fefbfb-420f-48d3-ade3-dede3da5b8e7","Type":"ContainerDied","Data":"b0c5e9641c82a20cdbdcb390e3be090df0efece3230889829254871c960dcefe"} Oct 30 20:31:57.933251 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:57.933225 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9389b9-jmjk8" Oct 30 20:31:58.059516 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:58.059446 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36fefbfb-420f-48d3-ade3-dede3da5b8e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36fefbfb-420f-48d3-ade3-dede3da5b8e7-maas-api-xks-ca\") pod \"36fefbfb-420f-48d3-ade3-dede3da5b8e7\" (UID: \"36fefbfb-420f-48d3-ade3-dede3da5b8e7\") " Oct 30 20:31:58.059516 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:58.059490 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36fefbfb-420f-48d3-ade3-dede3da5b8e7-kube-api-access-pcgrm\" (UniqueName: \"kubernetes.io/projected/36fefbfb-420f-48d3-ade3-dede3da5b8e7-kube-api-access-pcgrm\") pod \"36fefbfb-420f-48d3-ade3-dede3da5b8e7\" (UID: \"36fefbfb-420f-48d3-ade3-dede3da5b8e7\") " Oct 30 20:31:58.059737 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:58.059560 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36fefbfb-420f-48d3-ade3-dede3da5b8e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36fefbfb-420f-48d3-ade3-dede3da5b8e7-maas-api-openshift-ca\") pod \"36fefbfb-420f-48d3-ade3-dede3da5b8e7\" (UID: \"36fefbfb-420f-48d3-ade3-dede3da5b8e7\") " Oct 30 20:31:58.060014 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:58.059984 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36fefbfb-420f-48d3-ade3-dede3da5b8e7-maas-api-openshift-ca" pod "36fefbfb-420f-48d3-ade3-dede3da5b8e7" (UID: "36fefbfb-420f-48d3-ade3-dede3da5b8e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:31:58.061939 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:58.061910 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36fefbfb-420f-48d3-ade3-dede3da5b8e7-kube-api-access-pcgrm" pod "36fefbfb-420f-48d3-ade3-dede3da5b8e7" (UID: "36fefbfb-420f-48d3-ade3-dede3da5b8e7"). InnerVolumeSpecName "kube-api-access-pcgrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:31:58.062051 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:58.061978 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36fefbfb-420f-48d3-ade3-dede3da5b8e7-maas-api-xks-ca" pod "36fefbfb-420f-48d3-ade3-dede3da5b8e7" (UID: "36fefbfb-420f-48d3-ade3-dede3da5b8e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:31:58.161022 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:58.160980 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36fefbfb-420f-48d3-ade3-dede3da5b8e7-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:31:58.161022 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:58.161010 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcgrm\" (UniqueName: \"kubernetes.io/projected/36fefbfb-420f-48d3-ade3-dede3da5b8e7-kube-api-access-pcgrm\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:31:58.161022 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:58.161020 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36fefbfb-420f-48d3-ade3-dede3da5b8e7-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:31:58.780266 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:58.780236 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9389b9-jmjk8" Oct 30 20:31:58.780474 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:58.780270 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9389b9-jmjk8" event={"ID":"36fefbfb-420f-48d3-ade3-dede3da5b8e7","Type":"ContainerDied","Data":"570c8a868d88df2a76938b046491e9b9e5e2771ac4247180b1340d1e3ee54a98"} Oct 30 20:31:58.780474 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:31:58.780332 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="570c8a868d88df2a76938b046491e9b9e5e2771ac4247180b1340d1e3ee54a98" Oct 30 20:32:03.436467 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:03.436433 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9389b9-589ddf7656-4l682"] Oct 30 20:32:03.549044 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:03.549011 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9389b9-67dfc7496c-fq2ld"] Oct 30 20:32:05.455054 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:05.455020 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8d612c-zvjtl"] Oct 30 20:32:05.455557 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:05.455542 4292 cpu_manager.go:414] "containerMap: removing container" podUID="36fefbfb-420f-48d3-ade3-dede3da5b8e7" containerName="revoke-keys" Oct 30 20:32:05.455608 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:05.455558 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36fefbfb-420f-48d3-ade3-dede3da5b8e7" containerName="revoke-keys" Oct 30 20:32:05.455647 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:05.455638 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="36fefbfb-420f-48d3-ade3-dede3da5b8e7" containerName="revoke-keys" Oct 30 20:32:05.460275 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:05.460250 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8d612c-zvjtl" Oct 30 20:32:05.472308 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:05.472278 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ef191e2-03a6-4e19-8387-87817f09b447-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-8d612c-zvjtl\" (UID: \"5ef191e2-03a6-4e19-8387-87817f09b447\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8d612c-zvjtl" Oct 30 20:32:05.472456 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:05.472439 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ef191e2-03a6-4e19-8387-87817f09b447-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-8d612c-zvjtl\" (UID: \"5ef191e2-03a6-4e19-8387-87817f09b447\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8d612c-zvjtl" Oct 30 20:32:05.472501 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:05.472489 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft6dl\" (UniqueName: \"kubernetes.io/projected/5ef191e2-03a6-4e19-8387-87817f09b447-kube-api-access-ft6dl\") pod \"maas-api-revoke-keys-e2e-extmod-a-8d612c-zvjtl\" (UID: \"5ef191e2-03a6-4e19-8387-87817f09b447\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8d612c-zvjtl" Oct 30 20:32:05.513693 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:05.513669 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8d612c-zvjtl"] Oct 30 20:32:05.561410 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:05.561341 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:32:05.561560 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:05.561342 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\"" Oct 30 20:32:05.574719 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:05.574693 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ef191e2-03a6-4e19-8387-87817f09b447-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-8d612c-zvjtl\" (UID: \"5ef191e2-03a6-4e19-8387-87817f09b447\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8d612c-zvjtl" Oct 30 20:32:05.574838 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:05.574749 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ef191e2-03a6-4e19-8387-87817f09b447-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-8d612c-zvjtl\" (UID: \"5ef191e2-03a6-4e19-8387-87817f09b447\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8d612c-zvjtl" Oct 30 20:32:05.574838 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:05.574773 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft6dl\" (UniqueName: \"kubernetes.io/projected/5ef191e2-03a6-4e19-8387-87817f09b447-kube-api-access-ft6dl\") pod \"maas-api-revoke-keys-e2e-extmod-a-8d612c-zvjtl\" (UID: \"5ef191e2-03a6-4e19-8387-87817f09b447\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8d612c-zvjtl" Oct 30 20:32:05.575310 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:05.575290 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ef191e2-03a6-4e19-8387-87817f09b447-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-8d612c-zvjtl\" (UID: \"5ef191e2-03a6-4e19-8387-87817f09b447\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8d612c-zvjtl" Oct 30 20:32:05.577223 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:05.577202 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ef191e2-03a6-4e19-8387-87817f09b447-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-8d612c-zvjtl\" (UID: \"5ef191e2-03a6-4e19-8387-87817f09b447\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8d612c-zvjtl" Oct 30 20:32:05.637225 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:05.637193 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft6dl\" (UniqueName: \"kubernetes.io/projected/5ef191e2-03a6-4e19-8387-87817f09b447-kube-api-access-ft6dl\") pod \"maas-api-revoke-keys-e2e-extmod-a-8d612c-zvjtl\" (UID: \"5ef191e2-03a6-4e19-8387-87817f09b447\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8d612c-zvjtl" Oct 30 20:32:05.775663 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:05.775638 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8d612c-zvjtl" Oct 30 20:32:06.147076 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:32:06.147045 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ef191e2_03a6_4e19_8387_87817f09b447.slice/crio-628d5cc97b4e224eb8e3beaeb74385cd6c731051883ad57ae829e18db935ec42 WatchSource:0}: Error finding container 628d5cc97b4e224eb8e3beaeb74385cd6c731051883ad57ae829e18db935ec42: Status 404 returned error can't find the container with id 628d5cc97b4e224eb8e3beaeb74385cd6c731051883ad57ae829e18db935ec42 Oct 30 20:32:06.150931 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:06.150861 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8d612c-zvjtl"] Oct 30 20:32:06.827085 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:06.827052 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8d612c-zvjtl" event={"ID":"5ef191e2-03a6-4e19-8387-87817f09b447","Type":"ContainerStarted","Data":"e981e78e721670733cd356f164a92a565190bd8d2ee9ad4c889db13cc4575784"} Oct 30 20:32:06.827491 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:06.827103 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8d612c-zvjtl" event={"ID":"5ef191e2-03a6-4e19-8387-87817f09b447","Type":"ContainerStarted","Data":"628d5cc97b4e224eb8e3beaeb74385cd6c731051883ad57ae829e18db935ec42"} Oct 30 20:32:06.946699 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:06.946643 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8d612c-zvjtl" podStartSLOduration=1.946630044 podStartE2EDuration="1.946630044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:32:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:32:06.944887639 +0000 UTC m=+4030.463203094" watchObservedRunningTime="2026-10-30 20:32:06.946630044 +0000 UTC m=+4030.464945500" Oct 30 20:32:07.835167 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:07.835138 4292 generic.go:356] "Generic (PLEG): container finished" podID="5ef191e2-03a6-4e19-8387-87817f09b447" containerID="e981e78e721670733cd356f164a92a565190bd8d2ee9ad4c889db13cc4575784" exitCode=0 Oct 30 20:32:07.835618 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:07.835225 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8d612c-zvjtl" event={"ID":"5ef191e2-03a6-4e19-8387-87817f09b447","Type":"ContainerDied","Data":"e981e78e721670733cd356f164a92a565190bd8d2ee9ad4c889db13cc4575784"} Oct 30 20:32:08.437124 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:08.437068 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-9389b9-589ddf7656-4l682" podUID="ae3a4829-e22b-4f59-a1f8-70ce77384238" containerName="payload-processing" containerID="cri-o://f92eae1bf5c2bc8266ac9d9a0933689a56f35c64e626eb6ea7351590251d3b30" gracePeriod=25 Oct 30 20:32:08.549813 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:08.549769 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9389b9-67dfc7496c-fq2ld" podUID="e13a90a4-daaf-462d-ab6a-7d2683c7bdc0" containerName="payload-pre-processing" containerID="cri-o://afcb7457e5f18a77a1ce153bb87228d93018b7e27c712917b2190453d0f52ca7" gracePeriod=25 Oct 30 20:32:08.709855 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:08.709828 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-9389b9-589ddf7656-4l682" Oct 30 20:32:08.811265 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:08.811243 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae3a4829-e22b-4f59-a1f8-70ce77384238-kube-api-access-792px\" (UniqueName: \"kubernetes.io/projected/ae3a4829-e22b-4f59-a1f8-70ce77384238-kube-api-access-792px\") pod \"ae3a4829-e22b-4f59-a1f8-70ce77384238\" (UID: \"ae3a4829-e22b-4f59-a1f8-70ce77384238\") " Oct 30 20:32:08.811444 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:08.811431 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae3a4829-e22b-4f59-a1f8-70ce77384238-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae3a4829-e22b-4f59-a1f8-70ce77384238-plugins-config-volume\") pod \"ae3a4829-e22b-4f59-a1f8-70ce77384238\" (UID: \"ae3a4829-e22b-4f59-a1f8-70ce77384238\") " Oct 30 20:32:08.811835 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:08.811811 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae3a4829-e22b-4f59-a1f8-70ce77384238-plugins-config-volume" pod "ae3a4829-e22b-4f59-a1f8-70ce77384238" (UID: "ae3a4829-e22b-4f59-a1f8-70ce77384238"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:32:08.813441 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:08.813415 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae3a4829-e22b-4f59-a1f8-70ce77384238-kube-api-access-792px" pod "ae3a4829-e22b-4f59-a1f8-70ce77384238" (UID: "ae3a4829-e22b-4f59-a1f8-70ce77384238"). InnerVolumeSpecName "kube-api-access-792px". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:32:08.816110 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:08.816094 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9389b9-67dfc7496c-fq2ld" Oct 30 20:32:08.844335 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:08.844304 4292 generic.go:356] "Generic (PLEG): container finished" podID="e13a90a4-daaf-462d-ab6a-7d2683c7bdc0" containerID="afcb7457e5f18a77a1ce153bb87228d93018b7e27c712917b2190453d0f52ca7" exitCode=0 Oct 30 20:32:08.844776 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:08.844376 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9389b9-67dfc7496c-fq2ld" Oct 30 20:32:08.844776 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:08.844386 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9389b9-67dfc7496c-fq2ld" event={"ID":"e13a90a4-daaf-462d-ab6a-7d2683c7bdc0","Type":"ContainerDied","Data":"afcb7457e5f18a77a1ce153bb87228d93018b7e27c712917b2190453d0f52ca7"} Oct 30 20:32:08.844776 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:08.844442 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9389b9-67dfc7496c-fq2ld" event={"ID":"e13a90a4-daaf-462d-ab6a-7d2683c7bdc0","Type":"ContainerDied","Data":"0f3ee086f8d7db527599df2bb05d54c49c5bb1dbc35a90a9ba7ee6b52858118c"} Oct 30 20:32:08.844776 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:08.844465 4292 scope.go:122] "RemoveContainer" containerID="afcb7457e5f18a77a1ce153bb87228d93018b7e27c712917b2190453d0f52ca7" Oct 30 20:32:08.846085 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:08.846064 4292 generic.go:356] "Generic (PLEG): container finished" podID="ae3a4829-e22b-4f59-a1f8-70ce77384238" containerID="f92eae1bf5c2bc8266ac9d9a0933689a56f35c64e626eb6ea7351590251d3b30" exitCode=0 Oct 30 20:32:08.846212 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:08.846103 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-9389b9-589ddf7656-4l682" Oct 30 20:32:08.846212 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:08.846196 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-9389b9-589ddf7656-4l682" event={"ID":"ae3a4829-e22b-4f59-a1f8-70ce77384238","Type":"ContainerDied","Data":"f92eae1bf5c2bc8266ac9d9a0933689a56f35c64e626eb6ea7351590251d3b30"} Oct 30 20:32:08.846345 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:08.846238 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-9389b9-589ddf7656-4l682" event={"ID":"ae3a4829-e22b-4f59-a1f8-70ce77384238","Type":"ContainerDied","Data":"4109f09d68145828f64a8866629e3a40c205ff22c627437268315f9ef57683ae"} Oct 30 20:32:08.857174 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:08.857149 4292 scope.go:122] "RemoveContainer" containerID="afcb7457e5f18a77a1ce153bb87228d93018b7e27c712917b2190453d0f52ca7" Oct 30 20:32:08.857592 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:32:08.857566 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afcb7457e5f18a77a1ce153bb87228d93018b7e27c712917b2190453d0f52ca7\": container with ID starting with afcb7457e5f18a77a1ce153bb87228d93018b7e27c712917b2190453d0f52ca7 not found: ID does not exist" containerID="afcb7457e5f18a77a1ce153bb87228d93018b7e27c712917b2190453d0f52ca7" Oct 30 20:32:08.857695 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:08.857602 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afcb7457e5f18a77a1ce153bb87228d93018b7e27c712917b2190453d0f52ca7"} err="failed to get container status \"afcb7457e5f18a77a1ce153bb87228d93018b7e27c712917b2190453d0f52ca7\": rpc error: code = NotFound desc = could not find container \"afcb7457e5f18a77a1ce153bb87228d93018b7e27c712917b2190453d0f52ca7\": container with ID starting with afcb7457e5f18a77a1ce153bb87228d93018b7e27c712917b2190453d0f52ca7 not found: ID does not exist" Oct 30 20:32:08.857695 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:08.857626 4292 scope.go:122] "RemoveContainer" containerID="f92eae1bf5c2bc8266ac9d9a0933689a56f35c64e626eb6ea7351590251d3b30" Oct 30 20:32:08.868204 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:08.868189 4292 scope.go:122] "RemoveContainer" containerID="f92eae1bf5c2bc8266ac9d9a0933689a56f35c64e626eb6ea7351590251d3b30" Oct 30 20:32:08.868597 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:32:08.868574 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f92eae1bf5c2bc8266ac9d9a0933689a56f35c64e626eb6ea7351590251d3b30\": container with ID starting with f92eae1bf5c2bc8266ac9d9a0933689a56f35c64e626eb6ea7351590251d3b30 not found: ID does not exist" containerID="f92eae1bf5c2bc8266ac9d9a0933689a56f35c64e626eb6ea7351590251d3b30" Oct 30 20:32:08.868697 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:08.868628 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f92eae1bf5c2bc8266ac9d9a0933689a56f35c64e626eb6ea7351590251d3b30"} err="failed to get container status \"f92eae1bf5c2bc8266ac9d9a0933689a56f35c64e626eb6ea7351590251d3b30\": rpc error: code = NotFound desc = could not find container \"f92eae1bf5c2bc8266ac9d9a0933689a56f35c64e626eb6ea7351590251d3b30\": container with ID starting with f92eae1bf5c2bc8266ac9d9a0933689a56f35c64e626eb6ea7351590251d3b30 not found: ID does not exist" Oct 30 20:32:08.912971 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:08.912934 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e13a90a4-daaf-462d-ab6a-7d2683c7bdc0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e13a90a4-daaf-462d-ab6a-7d2683c7bdc0-plugins-config-volume\") pod \"e13a90a4-daaf-462d-ab6a-7d2683c7bdc0\" (UID: \"e13a90a4-daaf-462d-ab6a-7d2683c7bdc0\") " Oct 30 20:32:08.913179 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:08.913026 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e13a90a4-daaf-462d-ab6a-7d2683c7bdc0-kube-api-access-rrvmr\" (UniqueName: \"kubernetes.io/projected/e13a90a4-daaf-462d-ab6a-7d2683c7bdc0-kube-api-access-rrvmr\") pod \"e13a90a4-daaf-462d-ab6a-7d2683c7bdc0\" (UID: \"e13a90a4-daaf-462d-ab6a-7d2683c7bdc0\") " Oct 30 20:32:08.913461 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:08.913439 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e13a90a4-daaf-462d-ab6a-7d2683c7bdc0-plugins-config-volume" pod "e13a90a4-daaf-462d-ab6a-7d2683c7bdc0" (UID: "e13a90a4-daaf-462d-ab6a-7d2683c7bdc0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:32:08.913646 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:08.913630 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-792px\" (UniqueName: \"kubernetes.io/projected/ae3a4829-e22b-4f59-a1f8-70ce77384238-kube-api-access-792px\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:32:08.913715 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:08.913652 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e13a90a4-daaf-462d-ab6a-7d2683c7bdc0-plugins-config-volume\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:32:08.913715 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:08.913668 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae3a4829-e22b-4f59-a1f8-70ce77384238-plugins-config-volume\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:32:08.915782 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:08.915755 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e13a90a4-daaf-462d-ab6a-7d2683c7bdc0-kube-api-access-rrvmr" pod "e13a90a4-daaf-462d-ab6a-7d2683c7bdc0" (UID: "e13a90a4-daaf-462d-ab6a-7d2683c7bdc0"). InnerVolumeSpecName "kube-api-access-rrvmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:32:08.970896 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:08.970864 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8d612c-zvjtl" Oct 30 20:32:09.014522 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:09.014497 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ef191e2-03a6-4e19-8387-87817f09b447-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ef191e2-03a6-4e19-8387-87817f09b447-maas-api-openshift-ca\") pod \"5ef191e2-03a6-4e19-8387-87817f09b447\" (UID: \"5ef191e2-03a6-4e19-8387-87817f09b447\") " Oct 30 20:32:09.014657 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:09.014532 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ef191e2-03a6-4e19-8387-87817f09b447-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ef191e2-03a6-4e19-8387-87817f09b447-maas-api-xks-ca\") pod \"5ef191e2-03a6-4e19-8387-87817f09b447\" (UID: \"5ef191e2-03a6-4e19-8387-87817f09b447\") " Oct 30 20:32:09.014712 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:09.014669 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ef191e2-03a6-4e19-8387-87817f09b447-kube-api-access-ft6dl\" (UniqueName: \"kubernetes.io/projected/5ef191e2-03a6-4e19-8387-87817f09b447-kube-api-access-ft6dl\") pod \"5ef191e2-03a6-4e19-8387-87817f09b447\" (UID: \"5ef191e2-03a6-4e19-8387-87817f09b447\") " Oct 30 20:32:09.014907 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:09.014887 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ef191e2-03a6-4e19-8387-87817f09b447-maas-api-openshift-ca" pod "5ef191e2-03a6-4e19-8387-87817f09b447" (UID: "5ef191e2-03a6-4e19-8387-87817f09b447"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:32:09.015094 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:09.015078 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ef191e2-03a6-4e19-8387-87817f09b447-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:32:09.015171 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:09.015096 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrvmr\" (UniqueName: \"kubernetes.io/projected/e13a90a4-daaf-462d-ab6a-7d2683c7bdc0-kube-api-access-rrvmr\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:32:09.016615 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:09.016590 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ef191e2-03a6-4e19-8387-87817f09b447-maas-api-xks-ca" pod "5ef191e2-03a6-4e19-8387-87817f09b447" (UID: "5ef191e2-03a6-4e19-8387-87817f09b447"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:32:09.016705 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:09.016640 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ef191e2-03a6-4e19-8387-87817f09b447-kube-api-access-ft6dl" pod "5ef191e2-03a6-4e19-8387-87817f09b447" (UID: "5ef191e2-03a6-4e19-8387-87817f09b447"). InnerVolumeSpecName "kube-api-access-ft6dl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:32:09.065691 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:09.065660 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9389b9-589ddf7656-4l682"] Oct 30 20:32:09.068614 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:09.068589 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9389b9-589ddf7656-4l682"] Oct 30 20:32:09.115882 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:09.115855 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ft6dl\" (UniqueName: \"kubernetes.io/projected/5ef191e2-03a6-4e19-8387-87817f09b447-kube-api-access-ft6dl\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:32:09.115882 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:09.115879 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ef191e2-03a6-4e19-8387-87817f09b447-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:32:09.356793 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:09.356766 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9389b9-67dfc7496c-fq2ld"] Oct 30 20:32:09.360929 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:09.360906 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9389b9-67dfc7496c-fq2ld"] Oct 30 20:32:09.854113 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:09.854084 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8d612c-zvjtl" Oct 30 20:32:09.854558 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:09.854120 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8d612c-zvjtl" event={"ID":"5ef191e2-03a6-4e19-8387-87817f09b447","Type":"ContainerDied","Data":"628d5cc97b4e224eb8e3beaeb74385cd6c731051883ad57ae829e18db935ec42"} Oct 30 20:32:09.854558 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:09.854166 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="628d5cc97b4e224eb8e3beaeb74385cd6c731051883ad57ae829e18db935ec42" Oct 30 20:32:10.581834 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:10.581804 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae3a4829-e22b-4f59-a1f8-70ce77384238" path="/var/lib/kubelet/pods/ae3a4829-e22b-4f59-a1f8-70ce77384238/volumes" Oct 30 20:32:10.582209 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:10.582196 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e13a90a4-daaf-462d-ab6a-7d2683c7bdc0" path="/var/lib/kubelet/pods/e13a90a4-daaf-462d-ab6a-7d2683c7bdc0/volumes" Oct 30 20:32:13.220330 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:32:13.220289 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:32:14.060066 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:14.060027 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af833e87-7f5w6"] Oct 30 20:32:14.111415 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:14.111219 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af833e87-7f5w6"] Oct 30 20:32:14.581815 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:14.581784 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac6c96de-884d-474f-84d7-8c58209986cd" path="/var/lib/kubelet/pods/ac6c96de-884d-474f-84d7-8c58209986cd/volumes" Oct 30 20:32:16.904062 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:16.904028 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-742ff8-t8jpt"] Oct 30 20:32:16.904719 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:16.904698 4292 cpu_manager.go:414] "containerMap: removing container" podUID="e13a90a4-daaf-462d-ab6a-7d2683c7bdc0" containerName="payload-pre-processing" Oct 30 20:32:16.904820 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:16.904719 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e13a90a4-daaf-462d-ab6a-7d2683c7bdc0" containerName="payload-pre-processing" Oct 30 20:32:16.904820 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:16.904743 4292 cpu_manager.go:414] "containerMap: removing container" podUID="5ef191e2-03a6-4e19-8387-87817f09b447" containerName="revoke-keys" Oct 30 20:32:16.904820 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:16.904753 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ef191e2-03a6-4e19-8387-87817f09b447" containerName="revoke-keys" Oct 30 20:32:16.904820 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:16.904791 4292 cpu_manager.go:414] "containerMap: removing container" podUID="ae3a4829-e22b-4f59-a1f8-70ce77384238" containerName="payload-processing" Oct 30 20:32:16.904820 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:16.904798 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae3a4829-e22b-4f59-a1f8-70ce77384238" containerName="payload-processing" Oct 30 20:32:16.905072 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:16.904927 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae3a4829-e22b-4f59-a1f8-70ce77384238" containerName="payload-processing" Oct 30 20:32:16.905072 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:16.904950 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="e13a90a4-daaf-462d-ab6a-7d2683c7bdc0" containerName="payload-pre-processing" Oct 30 20:32:16.905072 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:16.904963 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ef191e2-03a6-4e19-8387-87817f09b447" containerName="revoke-keys" Oct 30 20:32:16.909725 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:16.909704 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-742ff8-t8jpt" Oct 30 20:32:16.980378 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:16.979447 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-742ff8-t8jpt"] Oct 30 20:32:17.010296 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:17.010270 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:32:17.010296 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:17.010294 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\"" Oct 30 20:32:17.022801 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:17.022775 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tlm8\" (UniqueName: \"kubernetes.io/projected/04e30df5-4192-454b-ba91-ddfc5be5dfdf-kube-api-access-2tlm8\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-742ff8-t8jpt\" (UID: \"04e30df5-4192-454b-ba91-ddfc5be5dfdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-742ff8-t8jpt" Oct 30 20:32:17.022975 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:17.022962 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04e30df5-4192-454b-ba91-ddfc5be5dfdf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-742ff8-t8jpt\" (UID: \"04e30df5-4192-454b-ba91-ddfc5be5dfdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-742ff8-t8jpt" Oct 30 20:32:17.023116 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:17.023097 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04e30df5-4192-454b-ba91-ddfc5be5dfdf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-742ff8-t8jpt\" (UID: \"04e30df5-4192-454b-ba91-ddfc5be5dfdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-742ff8-t8jpt" Oct 30 20:32:17.123987 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:17.123954 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tlm8\" (UniqueName: \"kubernetes.io/projected/04e30df5-4192-454b-ba91-ddfc5be5dfdf-kube-api-access-2tlm8\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-742ff8-t8jpt\" (UID: \"04e30df5-4192-454b-ba91-ddfc5be5dfdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-742ff8-t8jpt" Oct 30 20:32:17.124166 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:17.123999 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04e30df5-4192-454b-ba91-ddfc5be5dfdf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-742ff8-t8jpt\" (UID: \"04e30df5-4192-454b-ba91-ddfc5be5dfdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-742ff8-t8jpt" Oct 30 20:32:17.124166 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:17.124038 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04e30df5-4192-454b-ba91-ddfc5be5dfdf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-742ff8-t8jpt\" (UID: \"04e30df5-4192-454b-ba91-ddfc5be5dfdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-742ff8-t8jpt" Oct 30 20:32:17.124763 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:17.124744 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04e30df5-4192-454b-ba91-ddfc5be5dfdf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-742ff8-t8jpt\" (UID: \"04e30df5-4192-454b-ba91-ddfc5be5dfdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-742ff8-t8jpt" Oct 30 20:32:17.126317 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:17.126293 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04e30df5-4192-454b-ba91-ddfc5be5dfdf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-742ff8-t8jpt\" (UID: \"04e30df5-4192-454b-ba91-ddfc5be5dfdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-742ff8-t8jpt" Oct 30 20:32:17.186844 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:17.186764 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tlm8\" (UniqueName: \"kubernetes.io/projected/04e30df5-4192-454b-ba91-ddfc5be5dfdf-kube-api-access-2tlm8\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-742ff8-t8jpt\" (UID: \"04e30df5-4192-454b-ba91-ddfc5be5dfdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-742ff8-t8jpt" Oct 30 20:32:17.222876 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:17.222845 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-742ff8-t8jpt" Oct 30 20:32:17.641086 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:32:17.638235 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04e30df5_4192_454b_ba91_ddfc5be5dfdf.slice/crio-9be0f24ba4b9a4433a03c06a48d06a481a9f2aa65b90a3907eaa4f6879b55015 WatchSource:0}: Error finding container 9be0f24ba4b9a4433a03c06a48d06a481a9f2aa65b90a3907eaa4f6879b55015: Status 404 returned error can't find the container with id 9be0f24ba4b9a4433a03c06a48d06a481a9f2aa65b90a3907eaa4f6879b55015 Oct 30 20:32:17.641086 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:17.641011 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-742ff8-t8jpt"] Oct 30 20:32:17.904007 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:17.903933 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-742ff8-t8jpt" event={"ID":"04e30df5-4192-454b-ba91-ddfc5be5dfdf","Type":"ContainerStarted","Data":"9be0f24ba4b9a4433a03c06a48d06a481a9f2aa65b90a3907eaa4f6879b55015"} Oct 30 20:32:18.361436 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:18.360640 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8d612c-74c797d987-9mbvg"] Oct 30 20:32:18.911673 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:18.911641 4292 generic.go:356] "Generic (PLEG): container finished" podID="04e30df5-4192-454b-ba91-ddfc5be5dfdf" containerID="538f721cfff19a1bac091250dffa8f0214b238565dbde21213de5d7ffe99d78e" exitCode=0 Oct 30 20:32:18.911856 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:18.911689 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-742ff8-t8jpt" event={"ID":"04e30df5-4192-454b-ba91-ddfc5be5dfdf","Type":"ContainerDied","Data":"538f721cfff19a1bac091250dffa8f0214b238565dbde21213de5d7ffe99d78e"} Oct 30 20:32:20.205823 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:20.205579 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-742ff8-t8jpt" Oct 30 20:32:20.276767 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:20.276730 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04e30df5-4192-454b-ba91-ddfc5be5dfdf-kube-api-access-2tlm8\" (UniqueName: \"kubernetes.io/projected/04e30df5-4192-454b-ba91-ddfc5be5dfdf-kube-api-access-2tlm8\") pod \"04e30df5-4192-454b-ba91-ddfc5be5dfdf\" (UID: \"04e30df5-4192-454b-ba91-ddfc5be5dfdf\") " Oct 30 20:32:20.276983 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:20.276785 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04e30df5-4192-454b-ba91-ddfc5be5dfdf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04e30df5-4192-454b-ba91-ddfc5be5dfdf-maas-api-openshift-ca\") pod \"04e30df5-4192-454b-ba91-ddfc5be5dfdf\" (UID: \"04e30df5-4192-454b-ba91-ddfc5be5dfdf\") " Oct 30 20:32:20.276983 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:20.276854 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04e30df5-4192-454b-ba91-ddfc5be5dfdf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04e30df5-4192-454b-ba91-ddfc5be5dfdf-maas-api-xks-ca\") pod \"04e30df5-4192-454b-ba91-ddfc5be5dfdf\" (UID: \"04e30df5-4192-454b-ba91-ddfc5be5dfdf\") " Oct 30 20:32:20.277235 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:20.277209 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04e30df5-4192-454b-ba91-ddfc5be5dfdf-maas-api-openshift-ca" pod "04e30df5-4192-454b-ba91-ddfc5be5dfdf" (UID: "04e30df5-4192-454b-ba91-ddfc5be5dfdf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:32:20.277442 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:20.277421 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04e30df5-4192-454b-ba91-ddfc5be5dfdf-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:32:20.279015 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:20.278985 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04e30df5-4192-454b-ba91-ddfc5be5dfdf-kube-api-access-2tlm8" pod "04e30df5-4192-454b-ba91-ddfc5be5dfdf" (UID: "04e30df5-4192-454b-ba91-ddfc5be5dfdf"). InnerVolumeSpecName "kube-api-access-2tlm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:32:20.279140 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:20.279014 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04e30df5-4192-454b-ba91-ddfc5be5dfdf-maas-api-xks-ca" pod "04e30df5-4192-454b-ba91-ddfc5be5dfdf" (UID: "04e30df5-4192-454b-ba91-ddfc5be5dfdf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:32:20.379043 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:20.379011 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tlm8\" (UniqueName: \"kubernetes.io/projected/04e30df5-4192-454b-ba91-ddfc5be5dfdf-kube-api-access-2tlm8\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:32:20.379043 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:20.379040 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04e30df5-4192-454b-ba91-ddfc5be5dfdf-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:32:20.927770 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:20.927689 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-742ff8-t8jpt" Oct 30 20:32:20.927770 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:20.927717 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-742ff8-t8jpt" event={"ID":"04e30df5-4192-454b-ba91-ddfc5be5dfdf","Type":"ContainerDied","Data":"9be0f24ba4b9a4433a03c06a48d06a481a9f2aa65b90a3907eaa4f6879b55015"} Oct 30 20:32:20.927770 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:20.927770 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9be0f24ba4b9a4433a03c06a48d06a481a9f2aa65b90a3907eaa4f6879b55015" Oct 30 20:32:23.361528 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:23.361483 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8d612c-74c797d987-9mbvg" podUID="73fcfe52-4127-40a3-a834-13200b4d2e85" containerName="payload-pre-processing" containerID="cri-o://baaf54e57cadce5f904e423993c955e5ae41c4481a5bd8e54b955f9bb8d00ceb" gracePeriod=25 Oct 30 20:32:23.459223 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:32:23.459193 4292 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c8e8826_ddee_49bf_8bbc_daa688aed93a.slice/crio-f4c4a4283ac98a8492f7b0ae07698ffee400affd9944fdc24894074039d622dc.scope\": RecentStats: unable to find data in memory cache]" Oct 30 20:32:23.609037 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:23.609014 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8d612c-74c797d987-9mbvg" Oct 30 20:32:23.727417 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:23.724090 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73fcfe52-4127-40a3-a834-13200b4d2e85-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73fcfe52-4127-40a3-a834-13200b4d2e85-plugins-config-volume\") pod \"73fcfe52-4127-40a3-a834-13200b4d2e85\" (UID: \"73fcfe52-4127-40a3-a834-13200b4d2e85\") " Oct 30 20:32:23.727417 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:23.724271 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73fcfe52-4127-40a3-a834-13200b4d2e85-kube-api-access-mqnzx\" (UniqueName: \"kubernetes.io/projected/73fcfe52-4127-40a3-a834-13200b4d2e85-kube-api-access-mqnzx\") pod \"73fcfe52-4127-40a3-a834-13200b4d2e85\" (UID: \"73fcfe52-4127-40a3-a834-13200b4d2e85\") " Oct 30 20:32:23.727417 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:23.725612 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73fcfe52-4127-40a3-a834-13200b4d2e85-plugins-config-volume" pod "73fcfe52-4127-40a3-a834-13200b4d2e85" (UID: "73fcfe52-4127-40a3-a834-13200b4d2e85"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:32:23.730331 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:23.730299 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73fcfe52-4127-40a3-a834-13200b4d2e85-kube-api-access-mqnzx" pod "73fcfe52-4127-40a3-a834-13200b4d2e85" (UID: "73fcfe52-4127-40a3-a834-13200b4d2e85"). InnerVolumeSpecName "kube-api-access-mqnzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:32:23.825437 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:23.825408 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqnzx\" (UniqueName: \"kubernetes.io/projected/73fcfe52-4127-40a3-a834-13200b4d2e85-kube-api-access-mqnzx\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:32:23.825437 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:23.825437 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73fcfe52-4127-40a3-a834-13200b4d2e85-plugins-config-volume\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:32:23.829933 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:23.829905 4292 patch_prober.go:29] interesting pod/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Oct 30 20:32:23.830058 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:23.829963 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz" podUID="2c8e8826-ddee-49bf-8bbc-daa688aed93a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Oct 30 20:32:23.947703 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:23.947669 4292 generic.go:356] "Generic (PLEG): container finished" podID="73fcfe52-4127-40a3-a834-13200b4d2e85" containerID="baaf54e57cadce5f904e423993c955e5ae41c4481a5bd8e54b955f9bb8d00ceb" exitCode=0 Oct 30 20:32:23.947909 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:23.947713 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8d612c-74c797d987-9mbvg" Oct 30 20:32:23.947909 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:23.947752 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8d612c-74c797d987-9mbvg" event={"ID":"73fcfe52-4127-40a3-a834-13200b4d2e85","Type":"ContainerDied","Data":"baaf54e57cadce5f904e423993c955e5ae41c4481a5bd8e54b955f9bb8d00ceb"} Oct 30 20:32:23.947909 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:23.947811 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8d612c-74c797d987-9mbvg" event={"ID":"73fcfe52-4127-40a3-a834-13200b4d2e85","Type":"ContainerDied","Data":"bb4c19f9cb5bf3203200ff87ce560b1a6ed19448125cd2f4d562fcc0980b3453"} Oct 30 20:32:23.947909 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:23.947840 4292 scope.go:122] "RemoveContainer" containerID="baaf54e57cadce5f904e423993c955e5ae41c4481a5bd8e54b955f9bb8d00ceb" Oct 30 20:32:23.949575 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:23.949547 4292 generic.go:356] "Generic (PLEG): container finished" podID="2c8e8826-ddee-49bf-8bbc-daa688aed93a" containerID="f4c4a4283ac98a8492f7b0ae07698ffee400affd9944fdc24894074039d622dc" exitCode=0 Oct 30 20:32:23.949697 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:23.949627 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz" event={"ID":"2c8e8826-ddee-49bf-8bbc-daa688aed93a","Type":"ContainerDied","Data":"f4c4a4283ac98a8492f7b0ae07698ffee400affd9944fdc24894074039d622dc"} Oct 30 20:32:23.950006 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:23.949993 4292 scope.go:122] "RemoveContainer" containerID="f4c4a4283ac98a8492f7b0ae07698ffee400affd9944fdc24894074039d622dc" Oct 30 20:32:23.951893 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:23.951863 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:32:23.952000 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:23.951951 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:32:23.952062 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:23.951997 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:32:23.959245 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:23.959229 4292 scope.go:122] "RemoveContainer" containerID="baaf54e57cadce5f904e423993c955e5ae41c4481a5bd8e54b955f9bb8d00ceb" Oct 30 20:32:23.959546 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:32:23.959527 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baaf54e57cadce5f904e423993c955e5ae41c4481a5bd8e54b955f9bb8d00ceb\": container with ID starting with baaf54e57cadce5f904e423993c955e5ae41c4481a5bd8e54b955f9bb8d00ceb not found: ID does not exist" containerID="baaf54e57cadce5f904e423993c955e5ae41c4481a5bd8e54b955f9bb8d00ceb" Oct 30 20:32:23.959612 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:23.959552 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baaf54e57cadce5f904e423993c955e5ae41c4481a5bd8e54b955f9bb8d00ceb"} err="failed to get container status \"baaf54e57cadce5f904e423993c955e5ae41c4481a5bd8e54b955f9bb8d00ceb\": rpc error: code = NotFound desc = could not find container \"baaf54e57cadce5f904e423993c955e5ae41c4481a5bd8e54b955f9bb8d00ceb\": container with ID starting with baaf54e57cadce5f904e423993c955e5ae41c4481a5bd8e54b955f9bb8d00ceb not found: ID does not exist" Oct 30 20:32:24.055364 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:24.055310 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb8de611-lrvmg"] Oct 30 20:32:24.057984 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:24.057959 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb8de611-lrvmg"] Oct 30 20:32:24.239878 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:24.239784 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8d612c-74c797d987-9mbvg"] Oct 30 20:32:24.244067 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:24.244037 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8d612c-74c797d987-9mbvg"] Oct 30 20:32:24.585673 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:24.585642 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="710fb1e6-2030-45fc-91b5-d4f13d6941b9" path="/var/lib/kubelet/pods/710fb1e6-2030-45fc-91b5-d4f13d6941b9/volumes" Oct 30 20:32:24.586088 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:24.586073 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73fcfe52-4127-40a3-a834-13200b4d2e85" path="/var/lib/kubelet/pods/73fcfe52-4127-40a3-a834-13200b4d2e85/volumes" Oct 30 20:32:24.980628 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:24.980533 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz" event={"ID":"2c8e8826-ddee-49bf-8bbc-daa688aed93a","Type":"ContainerStarted","Data":"0584ab7d07df9e9ab0b8057b09e825b48f1ca2a59a170150f8be8b65bee11318"} Oct 30 20:32:25.481704 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:25.481659 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz" Oct 30 20:32:25.481905 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:25.481861 4292 patch_prober.go:29] interesting pod/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Oct 30 20:32:25.481955 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:25.481915 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz" podUID="2c8e8826-ddee-49bf-8bbc-daa688aed93a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Oct 30 20:32:26.481704 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:26.481669 4292 patch_prober.go:29] interesting pod/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Oct 30 20:32:26.482113 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:26.481731 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz" podUID="2c8e8826-ddee-49bf-8bbc-daa688aed93a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Oct 30 20:32:27.481706 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:27.481666 4292 patch_prober.go:29] interesting pod/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Oct 30 20:32:27.482116 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:27.481733 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz" podUID="2c8e8826-ddee-49bf-8bbc-daa688aed93a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Oct 30 20:32:28.481338 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:28.481296 4292 patch_prober.go:29] interesting pod/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Oct 30 20:32:28.481553 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:28.481400 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz" podUID="2c8e8826-ddee-49bf-8bbc-daa688aed93a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Oct 30 20:32:29.481221 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:29.481184 4292 patch_prober.go:29] interesting pod/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Oct 30 20:32:29.481685 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:29.481241 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz" podUID="2c8e8826-ddee-49bf-8bbc-daa688aed93a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Oct 30 20:32:30.485407 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:30.485338 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz" Oct 30 20:32:30.485894 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:30.485811 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz" Oct 30 20:32:30.486612 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:30.486596 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz" Oct 30 20:32:38.051803 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:38.051768 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38554f47-lqw4b"] Oct 30 20:32:38.102537 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:38.102498 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38554f47-lqw4b"] Oct 30 20:32:38.582662 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:38.582632 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f18d86a4-efcc-4bb0-91c0-c98e1765acbb" path="/var/lib/kubelet/pods/f18d86a4-efcc-4bb0-91c0-c98e1765acbb/volumes" Oct 30 20:32:39.051676 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:39.051635 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48526fc3-g7k64"] Oct 30 20:32:39.102556 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:39.102521 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48526fc3-g7k64"] Oct 30 20:32:40.582308 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:40.582277 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f162b4dd-c258-4b49-8612-44b7a83122c7" path="/var/lib/kubelet/pods/f162b4dd-c258-4b49-8612-44b7a83122c7/volumes" Oct 30 20:32:43.220912 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:32:43.220875 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:32:49.139805 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:49.139767 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35ecd54c-2jb5d"] Oct 30 20:32:49.142279 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:49.142243 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dda0de5c-7pm6d"] Oct 30 20:32:49.144445 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:49.144430 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35ecd54c-2jb5d"] Oct 30 20:32:49.146433 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:49.146412 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dda0de5c-7pm6d"] Oct 30 20:32:50.582392 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:50.582333 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16d5a450-60c4-4075-8f0f-a5d628d7972d" path="/var/lib/kubelet/pods/16d5a450-60c4-4075-8f0f-a5d628d7972d/volumes" Oct 30 20:32:50.582819 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:50.582806 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ee60a2b-7f5f-493a-ad4a-bdd51ae6eefb" path="/var/lib/kubelet/pods/2ee60a2b-7f5f-493a-ad4a-bdd51ae6eefb/volumes" Oct 30 20:32:59.432661 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:59.432627 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e391de-7b69fb5d95-gtqdh"] Oct 30 20:32:59.433120 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:59.433105 4292 cpu_manager.go:414] "containerMap: removing container" podUID="73fcfe52-4127-40a3-a834-13200b4d2e85" containerName="payload-pre-processing" Oct 30 20:32:59.433173 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:59.433121 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73fcfe52-4127-40a3-a834-13200b4d2e85" containerName="payload-pre-processing" Oct 30 20:32:59.433173 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:59.433143 4292 cpu_manager.go:414] "containerMap: removing container" podUID="04e30df5-4192-454b-ba91-ddfc5be5dfdf" containerName="revoke-keys" Oct 30 20:32:59.433173 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:59.433154 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04e30df5-4192-454b-ba91-ddfc5be5dfdf" containerName="revoke-keys" Oct 30 20:32:59.433276 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:59.433247 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="73fcfe52-4127-40a3-a834-13200b4d2e85" containerName="payload-pre-processing" Oct 30 20:32:59.433276 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:59.433257 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="04e30df5-4192-454b-ba91-ddfc5be5dfdf" containerName="revoke-keys" Oct 30 20:32:59.436652 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:59.436635 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e391de-7b69fb5d95-gtqdh" Oct 30 20:32:59.530111 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:59.530074 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d0ec8d7-e0d4-4452-b263-89a3239af156-metrics-certs\") pod \"maas-api-e2e-extmod-b-e391de-7b69fb5d95-gtqdh\" (UID: \"9d0ec8d7-e0d4-4452-b263-89a3239af156\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e391de-7b69fb5d95-gtqdh" Oct 30 20:32:59.530298 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:59.530235 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d0ec8d7-e0d4-4452-b263-89a3239af156-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-e391de-7b69fb5d95-gtqdh\" (UID: \"9d0ec8d7-e0d4-4452-b263-89a3239af156\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e391de-7b69fb5d95-gtqdh" Oct 30 20:32:59.530298 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:59.530271 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22kdq\" (UniqueName: \"kubernetes.io/projected/9d0ec8d7-e0d4-4452-b263-89a3239af156-kube-api-access-22kdq\") pod \"maas-api-e2e-extmod-b-e391de-7b69fb5d95-gtqdh\" (UID: \"9d0ec8d7-e0d4-4452-b263-89a3239af156\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e391de-7b69fb5d95-gtqdh" Oct 30 20:32:59.530494 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:59.530299 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d0ec8d7-e0d4-4452-b263-89a3239af156-maas-api-tls\") pod \"maas-api-e2e-extmod-b-e391de-7b69fb5d95-gtqdh\" (UID: \"9d0ec8d7-e0d4-4452-b263-89a3239af156\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e391de-7b69fb5d95-gtqdh" Oct 30 20:32:59.550194 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:59.550171 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-e391de-serving-cert\"" Oct 30 20:32:59.553041 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:59.553012 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e391de-7b69fb5d95-gtqdh"] Oct 30 20:32:59.631192 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:59.631159 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d0ec8d7-e0d4-4452-b263-89a3239af156-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-e391de-7b69fb5d95-gtqdh\" (UID: \"9d0ec8d7-e0d4-4452-b263-89a3239af156\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e391de-7b69fb5d95-gtqdh" Oct 30 20:32:59.631394 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:59.631205 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22kdq\" (UniqueName: \"kubernetes.io/projected/9d0ec8d7-e0d4-4452-b263-89a3239af156-kube-api-access-22kdq\") pod \"maas-api-e2e-extmod-b-e391de-7b69fb5d95-gtqdh\" (UID: \"9d0ec8d7-e0d4-4452-b263-89a3239af156\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e391de-7b69fb5d95-gtqdh" Oct 30 20:32:59.631394 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:59.631241 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d0ec8d7-e0d4-4452-b263-89a3239af156-maas-api-tls\") pod \"maas-api-e2e-extmod-b-e391de-7b69fb5d95-gtqdh\" (UID: \"9d0ec8d7-e0d4-4452-b263-89a3239af156\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e391de-7b69fb5d95-gtqdh" Oct 30 20:32:59.631394 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:59.631328 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d0ec8d7-e0d4-4452-b263-89a3239af156-metrics-certs\") pod \"maas-api-e2e-extmod-b-e391de-7b69fb5d95-gtqdh\" (UID: \"9d0ec8d7-e0d4-4452-b263-89a3239af156\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e391de-7b69fb5d95-gtqdh" Oct 30 20:32:59.631572 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:32:59.631457 4292 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-e391de-serving-cert: secret "maas-api-e2e-extmod-b-e391de-serving-cert" not found Oct 30 20:32:59.631572 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:32:59.631540 4292 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d0ec8d7-e0d4-4452-b263-89a3239af156-maas-api-tls podName:9d0ec8d7-e0d4-4452-b263-89a3239af156 nodeName:}" failed. No retries permitted until 2026-10-30 20:33:00.131521106 +0000 UTC m=+4083.649836561 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9d0ec8d7-e0d4-4452-b263-89a3239af156-maas-api-tls") pod "maas-api-e2e-extmod-b-e391de-7b69fb5d95-gtqdh" (UID: "9d0ec8d7-e0d4-4452-b263-89a3239af156") : secret "maas-api-e2e-extmod-b-e391de-serving-cert" not found Oct 30 20:32:59.631851 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:59.631823 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d0ec8d7-e0d4-4452-b263-89a3239af156-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-e391de-7b69fb5d95-gtqdh\" (UID: \"9d0ec8d7-e0d4-4452-b263-89a3239af156\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e391de-7b69fb5d95-gtqdh" Oct 30 20:32:59.633712 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:59.633693 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d0ec8d7-e0d4-4452-b263-89a3239af156-metrics-certs\") pod \"maas-api-e2e-extmod-b-e391de-7b69fb5d95-gtqdh\" (UID: \"9d0ec8d7-e0d4-4452-b263-89a3239af156\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e391de-7b69fb5d95-gtqdh" Oct 30 20:32:59.693979 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:32:59.693910 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22kdq\" (UniqueName: \"kubernetes.io/projected/9d0ec8d7-e0d4-4452-b263-89a3239af156-kube-api-access-22kdq\") pod \"maas-api-e2e-extmod-b-e391de-7b69fb5d95-gtqdh\" (UID: \"9d0ec8d7-e0d4-4452-b263-89a3239af156\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e391de-7b69fb5d95-gtqdh" Oct 30 20:33:00.154271 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:00.154232 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d0ec8d7-e0d4-4452-b263-89a3239af156-maas-api-tls\") pod \"maas-api-e2e-extmod-b-e391de-7b69fb5d95-gtqdh\" (UID: \"9d0ec8d7-e0d4-4452-b263-89a3239af156\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e391de-7b69fb5d95-gtqdh" Oct 30 20:33:00.156704 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:00.156679 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d0ec8d7-e0d4-4452-b263-89a3239af156-maas-api-tls\") pod \"maas-api-e2e-extmod-b-e391de-7b69fb5d95-gtqdh\" (UID: \"9d0ec8d7-e0d4-4452-b263-89a3239af156\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e391de-7b69fb5d95-gtqdh" Oct 30 20:33:00.348558 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:00.348524 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e391de-7b69fb5d95-gtqdh" Oct 30 20:33:00.719438 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:33:00.719408 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d0ec8d7_e0d4_4452_b263_89a3239af156.slice/crio-ccc8af43c8c44cc14956d6f622ffe1ba713383e804fe9583e806c93fadba6966 WatchSource:0}: Error finding container ccc8af43c8c44cc14956d6f622ffe1ba713383e804fe9583e806c93fadba6966: Status 404 returned error can't find the container with id ccc8af43c8c44cc14956d6f622ffe1ba713383e804fe9583e806c93fadba6966 Oct 30 20:33:00.722431 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:00.722406 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e391de-7b69fb5d95-gtqdh"] Oct 30 20:33:01.201019 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:01.200979 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e391de-7b69fb5d95-gtqdh" event={"ID":"9d0ec8d7-e0d4-4452-b263-89a3239af156","Type":"ContainerStarted","Data":"ccc8af43c8c44cc14956d6f622ffe1ba713383e804fe9583e806c93fadba6966"} Oct 30 20:33:02.208225 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:02.208185 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e391de-7b69fb5d95-gtqdh" event={"ID":"9d0ec8d7-e0d4-4452-b263-89a3239af156","Type":"ContainerStarted","Data":"8670f3d3277c9ae30c884994daae95e4a0f3ad13640180ee2cc1a74d183d97f8"} Oct 30 20:33:02.208648 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:02.208283 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e391de-7b69fb5d95-gtqdh" Oct 30 20:33:02.331990 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:02.331933 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e391de-7b69fb5d95-gtqdh" podStartSLOduration=2.378637973 podStartE2EDuration="3.331920764s" totalImagesPullingTime="953.282791ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:32:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:33:02.329596757 +0000 UTC m=+4085.847912233" watchObservedRunningTime="2026-10-30 20:33:02.331920764 +0000 UTC m=+4085.850236220" Oct 30 20:33:08.219857 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:08.219831 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e391de-7b69fb5d95-gtqdh" Oct 30 20:33:13.221408 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:33:13.221273 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:33:32.425054 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:32.425024 4292 generic.go:356] "Generic (PLEG): container finished" podID="9d5226bf-9ab0-496b-b2f5-f0d730554365" containerID="ac68c6895d5768fa483ea1155501e73eb48cea63371569daa1ec1260f43328ba" exitCode=0 Oct 30 20:33:32.425564 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:32.425105 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms" event={"ID":"9d5226bf-9ab0-496b-b2f5-f0d730554365","Type":"ContainerDied","Data":"ac68c6895d5768fa483ea1155501e73eb48cea63371569daa1ec1260f43328ba"} Oct 30 20:33:32.425564 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:32.425553 4292 scope.go:122] "RemoveContainer" containerID="ac68c6895d5768fa483ea1155501e73eb48cea63371569daa1ec1260f43328ba" Oct 30 20:33:32.427116 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:32.427091 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:33:32.427200 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:32.427159 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:33:32.427200 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:32.427191 4292 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:33:33.433020 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:33.432983 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms" event={"ID":"9d5226bf-9ab0-496b-b2f5-f0d730554365","Type":"ContainerStarted","Data":"41e0a2b5abd2268e6d14c9a1757705f4d392117878465c05d42f78d09b9aeb25"} Oct 30 20:33:34.213248 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:34.213214 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms" Oct 30 20:33:34.213455 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:34.213438 4292 patch_prober.go:29] interesting pod/e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body= Oct 30 20:33:34.213531 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:34.213483 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms" podUID="9d5226bf-9ab0-496b-b2f5-f0d730554365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" Oct 30 20:33:35.212413 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:35.212375 4292 patch_prober.go:29] interesting pod/e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body= Oct 30 20:33:35.212870 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:35.212456 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms" podUID="9d5226bf-9ab0-496b-b2f5-f0d730554365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" Oct 30 20:33:36.212571 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:36.212540 4292 patch_prober.go:29] interesting pod/e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body= Oct 30 20:33:36.212972 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:36.212591 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms" podUID="9d5226bf-9ab0-496b-b2f5-f0d730554365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" Oct 30 20:33:37.212999 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:37.212963 4292 patch_prober.go:29] interesting pod/e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body= Oct 30 20:33:37.213460 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:37.213031 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms" podUID="9d5226bf-9ab0-496b-b2f5-f0d730554365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" Oct 30 20:33:38.213404 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:38.213337 4292 patch_prober.go:29] interesting pod/e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body= Oct 30 20:33:38.213789 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:38.213434 4292 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms" podUID="9d5226bf-9ab0-496b-b2f5-f0d730554365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" Oct 30 20:33:39.216996 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:39.216962 4292 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms" Oct 30 20:33:39.217544 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:39.217393 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms" Oct 30 20:33:39.218114 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:39.218098 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms" Oct 30 20:33:43.220793 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:33:43.220750 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:33:43.367339 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:43.367306 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9389b9-jmjk8"] Oct 30 20:33:43.417956 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:43.417915 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9389b9-jmjk8"] Oct 30 20:33:44.582107 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:44.582076 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36fefbfb-420f-48d3-ade3-dede3da5b8e7" path="/var/lib/kubelet/pods/36fefbfb-420f-48d3-ade3-dede3da5b8e7/volumes" Oct 30 20:33:49.881645 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:49.881607 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-742ff8-t8jpt"] Oct 30 20:33:49.932837 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:49.932800 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-742ff8-t8jpt"] Oct 30 20:33:49.988483 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:49.988449 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz"] Oct 30 20:33:49.988750 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:49.988723 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz" podUID="2c8e8826-ddee-49bf-8bbc-daa688aed93a" containerName="istio-proxy" containerID="cri-o://0584ab7d07df9e9ab0b8057b09e825b48f1ca2a59a170150f8be8b65bee11318" gracePeriod=30 Oct 30 20:33:50.583596 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:50.583565 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04e30df5-4192-454b-ba91-ddfc5be5dfdf" path="/var/lib/kubelet/pods/04e30df5-4192-454b-ba91-ddfc5be5dfdf/volumes" Oct 30 20:33:51.150671 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:51.150636 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8d612c-zvjtl"] Oct 30 20:33:51.201296 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:51.201263 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8d612c-zvjtl"] Oct 30 20:33:52.590558 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:52.590520 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ef191e2-03a6-4e19-8387-87817f09b447" path="/var/lib/kubelet/pods/5ef191e2-03a6-4e19-8387-87817f09b447/volumes" Oct 30 20:33:55.241391 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:55.241328 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz" Oct 30 20:33:55.342824 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:55.342791 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c8e8826-ddee-49bf-8bbc-daa688aed93a-kube-api-access-m929d\" (UniqueName: \"kubernetes.io/projected/2c8e8826-ddee-49bf-8bbc-daa688aed93a-kube-api-access-m929d\") pod \"2c8e8826-ddee-49bf-8bbc-daa688aed93a\" (UID: \"2c8e8826-ddee-49bf-8bbc-daa688aed93a\") " Oct 30 20:33:55.342975 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:55.342840 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c8e8826-ddee-49bf-8bbc-daa688aed93a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c8e8826-ddee-49bf-8bbc-daa688aed93a-workload-certs\") pod \"2c8e8826-ddee-49bf-8bbc-daa688aed93a\" (UID: \"2c8e8826-ddee-49bf-8bbc-daa688aed93a\") " Oct 30 20:33:55.342975 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:55.342946 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c8e8826-ddee-49bf-8bbc-daa688aed93a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c8e8826-ddee-49bf-8bbc-daa688aed93a-istio-data\") pod \"2c8e8826-ddee-49bf-8bbc-daa688aed93a\" (UID: \"2c8e8826-ddee-49bf-8bbc-daa688aed93a\") " Oct 30 20:33:55.343057 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:55.342984 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c8e8826-ddee-49bf-8bbc-daa688aed93a-istio-token\" (UniqueName: \"kubernetes.io/projected/2c8e8826-ddee-49bf-8bbc-daa688aed93a-istio-token\") pod \"2c8e8826-ddee-49bf-8bbc-daa688aed93a\" (UID: \"2c8e8826-ddee-49bf-8bbc-daa688aed93a\") " Oct 30 20:33:55.343109 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:55.343055 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c8e8826-ddee-49bf-8bbc-daa688aed93a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c8e8826-ddee-49bf-8bbc-daa688aed93a-istio-envoy\") pod \"2c8e8826-ddee-49bf-8bbc-daa688aed93a\" (UID: \"2c8e8826-ddee-49bf-8bbc-daa688aed93a\") " Oct 30 20:33:55.343163 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:55.343131 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c8e8826-ddee-49bf-8bbc-daa688aed93a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c8e8826-ddee-49bf-8bbc-daa688aed93a-istiod-ca-cert\") pod \"2c8e8826-ddee-49bf-8bbc-daa688aed93a\" (UID: \"2c8e8826-ddee-49bf-8bbc-daa688aed93a\") " Oct 30 20:33:55.343163 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:55.343136 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c8e8826-ddee-49bf-8bbc-daa688aed93a-workload-certs" pod "2c8e8826-ddee-49bf-8bbc-daa688aed93a" (UID: "2c8e8826-ddee-49bf-8bbc-daa688aed93a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:33:55.343286 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:55.343172 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c8e8826-ddee-49bf-8bbc-daa688aed93a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c8e8826-ddee-49bf-8bbc-daa688aed93a-istio-podinfo\") pod \"2c8e8826-ddee-49bf-8bbc-daa688aed93a\" (UID: \"2c8e8826-ddee-49bf-8bbc-daa688aed93a\") " Oct 30 20:33:55.343286 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:55.343180 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c8e8826-ddee-49bf-8bbc-daa688aed93a-istio-data" pod "2c8e8826-ddee-49bf-8bbc-daa688aed93a" (UID: "2c8e8826-ddee-49bf-8bbc-daa688aed93a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:33:55.343286 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:55.343217 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c8e8826-ddee-49bf-8bbc-daa688aed93a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c8e8826-ddee-49bf-8bbc-daa688aed93a-credential-socket\") pod \"2c8e8826-ddee-49bf-8bbc-daa688aed93a\" (UID: \"2c8e8826-ddee-49bf-8bbc-daa688aed93a\") " Oct 30 20:33:55.343286 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:55.343247 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c8e8826-ddee-49bf-8bbc-daa688aed93a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c8e8826-ddee-49bf-8bbc-daa688aed93a-workload-socket\") pod \"2c8e8826-ddee-49bf-8bbc-daa688aed93a\" (UID: \"2c8e8826-ddee-49bf-8bbc-daa688aed93a\") " Oct 30 20:33:55.343542 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:55.343520 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c8e8826-ddee-49bf-8bbc-daa688aed93a-credential-socket" pod "2c8e8826-ddee-49bf-8bbc-daa688aed93a" (UID: "2c8e8826-ddee-49bf-8bbc-daa688aed93a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:33:55.343606 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:55.343553 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c8e8826-ddee-49bf-8bbc-daa688aed93a-istiod-ca-cert" pod "2c8e8826-ddee-49bf-8bbc-daa688aed93a" (UID: "2c8e8826-ddee-49bf-8bbc-daa688aed93a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:33:55.343725 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:55.343705 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c8e8826-ddee-49bf-8bbc-daa688aed93a-workload-socket" pod "2c8e8826-ddee-49bf-8bbc-daa688aed93a" (UID: "2c8e8826-ddee-49bf-8bbc-daa688aed93a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:33:55.343856 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:55.343833 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c8e8826-ddee-49bf-8bbc-daa688aed93a-workload-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:33:55.343856 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:55.343854 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c8e8826-ddee-49bf-8bbc-daa688aed93a-istio-data\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:33:55.344055 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:55.343863 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c8e8826-ddee-49bf-8bbc-daa688aed93a-istiod-ca-cert\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:33:55.344055 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:55.343872 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c8e8826-ddee-49bf-8bbc-daa688aed93a-credential-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:33:55.344055 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:55.343880 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c8e8826-ddee-49bf-8bbc-daa688aed93a-workload-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:33:55.345206 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:55.345178 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c8e8826-ddee-49bf-8bbc-daa688aed93a-istio-envoy" pod "2c8e8826-ddee-49bf-8bbc-daa688aed93a" (UID: "2c8e8826-ddee-49bf-8bbc-daa688aed93a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:33:55.345300 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:55.345218 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c8e8826-ddee-49bf-8bbc-daa688aed93a-istio-token" pod "2c8e8826-ddee-49bf-8bbc-daa688aed93a" (UID: "2c8e8826-ddee-49bf-8bbc-daa688aed93a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:33:55.345469 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:55.345455 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c8e8826-ddee-49bf-8bbc-daa688aed93a-kube-api-access-m929d" pod "2c8e8826-ddee-49bf-8bbc-daa688aed93a" (UID: "2c8e8826-ddee-49bf-8bbc-daa688aed93a"). InnerVolumeSpecName "kube-api-access-m929d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:33:55.345542 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:55.345472 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c8e8826-ddee-49bf-8bbc-daa688aed93a-istio-podinfo" pod "2c8e8826-ddee-49bf-8bbc-daa688aed93a" (UID: "2c8e8826-ddee-49bf-8bbc-daa688aed93a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:33:55.444737 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:55.444709 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m929d\" (UniqueName: \"kubernetes.io/projected/2c8e8826-ddee-49bf-8bbc-daa688aed93a-kube-api-access-m929d\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:33:55.444737 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:55.444735 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c8e8826-ddee-49bf-8bbc-daa688aed93a-istio-token\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:33:55.444928 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:55.444744 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c8e8826-ddee-49bf-8bbc-daa688aed93a-istio-envoy\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:33:55.444928 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:55.444757 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c8e8826-ddee-49bf-8bbc-daa688aed93a-istio-podinfo\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:33:55.621835 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:55.621756 4292 generic.go:356] "Generic (PLEG): container finished" podID="2c8e8826-ddee-49bf-8bbc-daa688aed93a" containerID="0584ab7d07df9e9ab0b8057b09e825b48f1ca2a59a170150f8be8b65bee11318" exitCode=0 Oct 30 20:33:55.621835 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:55.621803 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz" Oct 30 20:33:55.621835 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:55.621809 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz" event={"ID":"2c8e8826-ddee-49bf-8bbc-daa688aed93a","Type":"ContainerDied","Data":"0584ab7d07df9e9ab0b8057b09e825b48f1ca2a59a170150f8be8b65bee11318"} Oct 30 20:33:55.622100 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:55.621857 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz" event={"ID":"2c8e8826-ddee-49bf-8bbc-daa688aed93a","Type":"ContainerDied","Data":"70ac8c916827646e8fb937eec104f97502e7b40962c23b8b71dbade02e96814a"} Oct 30 20:33:55.622100 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:55.621882 4292 scope.go:122] "RemoveContainer" containerID="0584ab7d07df9e9ab0b8057b09e825b48f1ca2a59a170150f8be8b65bee11318" Oct 30 20:33:55.632621 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:55.632601 4292 scope.go:122] "RemoveContainer" containerID="f4c4a4283ac98a8492f7b0ae07698ffee400affd9944fdc24894074039d622dc" Oct 30 20:33:55.641507 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:55.641487 4292 scope.go:122] "RemoveContainer" containerID="0584ab7d07df9e9ab0b8057b09e825b48f1ca2a59a170150f8be8b65bee11318" Oct 30 20:33:55.641786 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:33:55.641764 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0584ab7d07df9e9ab0b8057b09e825b48f1ca2a59a170150f8be8b65bee11318\": container with ID starting with 0584ab7d07df9e9ab0b8057b09e825b48f1ca2a59a170150f8be8b65bee11318 not found: ID does not exist" containerID="0584ab7d07df9e9ab0b8057b09e825b48f1ca2a59a170150f8be8b65bee11318" Oct 30 20:33:55.641863 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:55.641801 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0584ab7d07df9e9ab0b8057b09e825b48f1ca2a59a170150f8be8b65bee11318"} err="failed to get container status \"0584ab7d07df9e9ab0b8057b09e825b48f1ca2a59a170150f8be8b65bee11318\": rpc error: code = NotFound desc = could not find container \"0584ab7d07df9e9ab0b8057b09e825b48f1ca2a59a170150f8be8b65bee11318\": container with ID starting with 0584ab7d07df9e9ab0b8057b09e825b48f1ca2a59a170150f8be8b65bee11318 not found: ID does not exist" Oct 30 20:33:55.641863 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:55.641826 4292 scope.go:122] "RemoveContainer" containerID="f4c4a4283ac98a8492f7b0ae07698ffee400affd9944fdc24894074039d622dc" Oct 30 20:33:55.642105 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:33:55.642084 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4c4a4283ac98a8492f7b0ae07698ffee400affd9944fdc24894074039d622dc\": container with ID starting with f4c4a4283ac98a8492f7b0ae07698ffee400affd9944fdc24894074039d622dc not found: ID does not exist" containerID="f4c4a4283ac98a8492f7b0ae07698ffee400affd9944fdc24894074039d622dc" Oct 30 20:33:55.642146 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:55.642113 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4c4a4283ac98a8492f7b0ae07698ffee400affd9944fdc24894074039d622dc"} err="failed to get container status \"f4c4a4283ac98a8492f7b0ae07698ffee400affd9944fdc24894074039d622dc\": rpc error: code = NotFound desc = could not find container \"f4c4a4283ac98a8492f7b0ae07698ffee400affd9944fdc24894074039d622dc\": container with ID starting with f4c4a4283ac98a8492f7b0ae07698ffee400affd9944fdc24894074039d622dc not found: ID does not exist" Oct 30 20:33:55.791459 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:55.791426 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz"] Oct 30 20:33:55.842197 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:55.842166 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-79c649-openshift-default-756c8b9d99-cf2pz"] Oct 30 20:33:56.582601 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:56.582557 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c8e8826-ddee-49bf-8bbc-daa688aed93a" path="/var/lib/kubelet/pods/2c8e8826-ddee-49bf-8bbc-daa688aed93a/volumes" Oct 30 20:33:57.260523 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:57.260476 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl"] Oct 30 20:33:57.260808 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:57.260779 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl" podUID="7758bd21-5416-4ab0-8c70-59970dc01209" containerName="istio-proxy" containerID="cri-o://3d02355f58c54ce2360837abf8ca61b881af6f3d03c70941b2712da765dc8999" gracePeriod=30 Oct 30 20:33:58.120187 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:58.120159 4292 scope.go:122] "RemoveContainer" containerID="f8feed4dcc691609a029f529adc80f22f0fb1c03e9f0324d5b6b2b01cd938d46" Oct 30 20:33:58.131317 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:58.131297 4292 scope.go:122] "RemoveContainer" containerID="2a4c87d74b2a295d5a436a0d4bed89d31934944069d4db8567e178cb065793a0" Oct 30 20:33:58.139909 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:58.139891 4292 scope.go:122] "RemoveContainer" containerID="bec0195d457782ff6a7ec8d79bc348db47ca2fddc6796cacdea78dc34d74bf90" Oct 30 20:33:58.148992 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:58.148964 4292 scope.go:122] "RemoveContainer" containerID="aa99f136bf5eca6178981da3f088eab91d52360162828584f13b2d9b891960a8" Oct 30 20:33:58.158887 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:58.158865 4292 scope.go:122] "RemoveContainer" containerID="8a0fd5add8c6104c2c2771368e0ad9ab985e2cfecfa272d3d285419d2a921593" Oct 30 20:33:58.167052 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:58.167033 4292 scope.go:122] "RemoveContainer" containerID="d6647ca481fcc183fe8808378d5fde26761af095cfa501006d4af20417da7927" Oct 30 20:33:59.103229 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:59.103200 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d"] Oct 30 20:33:59.103554 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:33:59.103449 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d" podUID="476ee650-d28b-4580-9eb4-a9f095df7822" containerName="istio-proxy" containerID="cri-o://45750ddbb4237d69d75e88edb1ec8d5d39c3fc2f221cdc744cabf0ad758416f6" gracePeriod=30 Oct 30 20:34:01.056489 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:01.056458 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccbc8582-ljr7s"] Oct 30 20:34:01.108760 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:01.108727 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccbc8582-ljr7s"] Oct 30 20:34:01.362583 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:01.362490 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cafbb2-nhj6g"] Oct 30 20:34:01.362983 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:01.362971 4292 cpu_manager.go:414] "containerMap: removing container" podUID="2c8e8826-ddee-49bf-8bbc-daa688aed93a" containerName="istio-proxy" Oct 30 20:34:01.363039 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:01.362984 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c8e8826-ddee-49bf-8bbc-daa688aed93a" containerName="istio-proxy" Oct 30 20:34:01.363039 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:01.363005 4292 cpu_manager.go:414] "containerMap: removing container" podUID="2c8e8826-ddee-49bf-8bbc-daa688aed93a" containerName="istio-proxy" Oct 30 20:34:01.363039 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:01.363010 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c8e8826-ddee-49bf-8bbc-daa688aed93a" containerName="istio-proxy" Oct 30 20:34:01.363150 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:01.363120 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c8e8826-ddee-49bf-8bbc-daa688aed93a" containerName="istio-proxy" Oct 30 20:34:01.363150 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:01.363130 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c8e8826-ddee-49bf-8bbc-daa688aed93a" containerName="istio-proxy" Oct 30 20:34:01.366539 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:01.366518 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cafbb2-nhj6g" Oct 30 20:34:01.421139 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:01.421112 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cafbb2-nhj6g"] Oct 30 20:34:01.426827 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:01.426766 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:34:01.426990 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:01.426976 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\"" Oct 30 20:34:01.433562 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:01.433537 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5379527-3946-433f-8abb-a0d43cc70cde-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-cafbb2-nhj6g\" (UID: \"a5379527-3946-433f-8abb-a0d43cc70cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cafbb2-nhj6g" Oct 30 20:34:01.433716 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:01.433700 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5379527-3946-433f-8abb-a0d43cc70cde-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-cafbb2-nhj6g\" (UID: \"a5379527-3946-433f-8abb-a0d43cc70cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cafbb2-nhj6g" Oct 30 20:34:01.433764 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:01.433731 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp8vl\" (UniqueName: \"kubernetes.io/projected/a5379527-3946-433f-8abb-a0d43cc70cde-kube-api-access-dp8vl\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-cafbb2-nhj6g\" (UID: \"a5379527-3946-433f-8abb-a0d43cc70cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cafbb2-nhj6g" Oct 30 20:34:01.535003 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:01.534972 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5379527-3946-433f-8abb-a0d43cc70cde-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-cafbb2-nhj6g\" (UID: \"a5379527-3946-433f-8abb-a0d43cc70cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cafbb2-nhj6g" Oct 30 20:34:01.535211 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:01.535010 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp8vl\" (UniqueName: \"kubernetes.io/projected/a5379527-3946-433f-8abb-a0d43cc70cde-kube-api-access-dp8vl\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-cafbb2-nhj6g\" (UID: \"a5379527-3946-433f-8abb-a0d43cc70cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cafbb2-nhj6g" Oct 30 20:34:01.535211 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:01.535071 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5379527-3946-433f-8abb-a0d43cc70cde-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-cafbb2-nhj6g\" (UID: \"a5379527-3946-433f-8abb-a0d43cc70cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cafbb2-nhj6g" Oct 30 20:34:01.535667 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:01.535640 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5379527-3946-433f-8abb-a0d43cc70cde-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-cafbb2-nhj6g\" (UID: \"a5379527-3946-433f-8abb-a0d43cc70cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cafbb2-nhj6g" Oct 30 20:34:01.537431 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:01.537409 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5379527-3946-433f-8abb-a0d43cc70cde-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-cafbb2-nhj6g\" (UID: \"a5379527-3946-433f-8abb-a0d43cc70cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cafbb2-nhj6g" Oct 30 20:34:01.597503 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:01.597470 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp8vl\" (UniqueName: \"kubernetes.io/projected/a5379527-3946-433f-8abb-a0d43cc70cde-kube-api-access-dp8vl\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-cafbb2-nhj6g\" (UID: \"a5379527-3946-433f-8abb-a0d43cc70cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cafbb2-nhj6g" Oct 30 20:34:01.678257 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:01.678175 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cafbb2-nhj6g" Oct 30 20:34:02.060110 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:02.060085 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cafbb2-nhj6g"] Oct 30 20:34:02.061732 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:34:02.061700 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5379527_3946_433f_8abb_a0d43cc70cde.slice/crio-ef7d03f72a0b181c4ce10a7cf6b396cb58049fd6b1c4ee1de968b5c9f5e6ca22 WatchSource:0}: Error finding container ef7d03f72a0b181c4ce10a7cf6b396cb58049fd6b1c4ee1de968b5c9f5e6ca22: Status 404 returned error can't find the container with id ef7d03f72a0b181c4ce10a7cf6b396cb58049fd6b1c4ee1de968b5c9f5e6ca22 Oct 30 20:34:02.503804 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:02.503781 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl" Oct 30 20:34:02.582294 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:02.582266 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65afac6e-94e2-4531-aba0-0355ec17aeb7" path="/var/lib/kubelet/pods/65afac6e-94e2-4531-aba0-0355ec17aeb7/volumes" Oct 30 20:34:02.650889 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:02.650811 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7758bd21-5416-4ab0-8c70-59970dc01209-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7758bd21-5416-4ab0-8c70-59970dc01209-istiod-ca-cert\") pod \"7758bd21-5416-4ab0-8c70-59970dc01209\" (UID: \"7758bd21-5416-4ab0-8c70-59970dc01209\") " Oct 30 20:34:02.650889 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:02.650859 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7758bd21-5416-4ab0-8c70-59970dc01209-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7758bd21-5416-4ab0-8c70-59970dc01209-workload-socket\") pod \"7758bd21-5416-4ab0-8c70-59970dc01209\" (UID: \"7758bd21-5416-4ab0-8c70-59970dc01209\") " Oct 30 20:34:02.650889 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:02.650885 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7758bd21-5416-4ab0-8c70-59970dc01209-istio-token\" (UniqueName: \"kubernetes.io/projected/7758bd21-5416-4ab0-8c70-59970dc01209-istio-token\") pod \"7758bd21-5416-4ab0-8c70-59970dc01209\" (UID: \"7758bd21-5416-4ab0-8c70-59970dc01209\") " Oct 30 20:34:02.651148 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:02.650909 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7758bd21-5416-4ab0-8c70-59970dc01209-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7758bd21-5416-4ab0-8c70-59970dc01209-credential-socket\") pod \"7758bd21-5416-4ab0-8c70-59970dc01209\" (UID: \"7758bd21-5416-4ab0-8c70-59970dc01209\") " Oct 30 20:34:02.651148 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:02.651047 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7758bd21-5416-4ab0-8c70-59970dc01209-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7758bd21-5416-4ab0-8c70-59970dc01209-istio-podinfo\") pod \"7758bd21-5416-4ab0-8c70-59970dc01209\" (UID: \"7758bd21-5416-4ab0-8c70-59970dc01209\") " Oct 30 20:34:02.651148 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:02.651099 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7758bd21-5416-4ab0-8c70-59970dc01209-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7758bd21-5416-4ab0-8c70-59970dc01209-istio-data\") pod \"7758bd21-5416-4ab0-8c70-59970dc01209\" (UID: \"7758bd21-5416-4ab0-8c70-59970dc01209\") " Oct 30 20:34:02.651148 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:02.651141 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7758bd21-5416-4ab0-8c70-59970dc01209-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7758bd21-5416-4ab0-8c70-59970dc01209-istio-envoy\") pod \"7758bd21-5416-4ab0-8c70-59970dc01209\" (UID: \"7758bd21-5416-4ab0-8c70-59970dc01209\") " Oct 30 20:34:02.651396 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:02.651182 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7758bd21-5416-4ab0-8c70-59970dc01209-kube-api-access-z7ms2\" (UniqueName: \"kubernetes.io/projected/7758bd21-5416-4ab0-8c70-59970dc01209-kube-api-access-z7ms2\") pod \"7758bd21-5416-4ab0-8c70-59970dc01209\" (UID: \"7758bd21-5416-4ab0-8c70-59970dc01209\") " Oct 30 20:34:02.651396 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:02.651187 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7758bd21-5416-4ab0-8c70-59970dc01209-workload-socket" pod "7758bd21-5416-4ab0-8c70-59970dc01209" (UID: "7758bd21-5416-4ab0-8c70-59970dc01209"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:34:02.651396 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:02.651215 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7758bd21-5416-4ab0-8c70-59970dc01209-istiod-ca-cert" pod "7758bd21-5416-4ab0-8c70-59970dc01209" (UID: "7758bd21-5416-4ab0-8c70-59970dc01209"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:34:02.651396 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:02.651228 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7758bd21-5416-4ab0-8c70-59970dc01209-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7758bd21-5416-4ab0-8c70-59970dc01209-workload-certs\") pod \"7758bd21-5416-4ab0-8c70-59970dc01209\" (UID: \"7758bd21-5416-4ab0-8c70-59970dc01209\") " Oct 30 20:34:02.651636 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:02.651345 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7758bd21-5416-4ab0-8c70-59970dc01209-credential-socket" pod "7758bd21-5416-4ab0-8c70-59970dc01209" (UID: "7758bd21-5416-4ab0-8c70-59970dc01209"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:34:02.651636 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:02.651431 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7758bd21-5416-4ab0-8c70-59970dc01209-istio-data" pod "7758bd21-5416-4ab0-8c70-59970dc01209" (UID: "7758bd21-5416-4ab0-8c70-59970dc01209"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:34:02.651636 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:02.651612 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7758bd21-5416-4ab0-8c70-59970dc01209-workload-certs" pod "7758bd21-5416-4ab0-8c70-59970dc01209" (UID: "7758bd21-5416-4ab0-8c70-59970dc01209"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:34:02.651853 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:02.651669 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7758bd21-5416-4ab0-8c70-59970dc01209-istio-data\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:34:02.651853 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:02.651692 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7758bd21-5416-4ab0-8c70-59970dc01209-istiod-ca-cert\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:34:02.651853 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:02.651706 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7758bd21-5416-4ab0-8c70-59970dc01209-workload-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:34:02.651853 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:02.651719 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7758bd21-5416-4ab0-8c70-59970dc01209-credential-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:34:02.653520 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:02.653493 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7758bd21-5416-4ab0-8c70-59970dc01209-istio-envoy" pod "7758bd21-5416-4ab0-8c70-59970dc01209" (UID: "7758bd21-5416-4ab0-8c70-59970dc01209"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:34:02.653520 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:02.653498 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7758bd21-5416-4ab0-8c70-59970dc01209-istio-podinfo" pod "7758bd21-5416-4ab0-8c70-59970dc01209" (UID: "7758bd21-5416-4ab0-8c70-59970dc01209"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:34:02.653815 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:02.653793 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7758bd21-5416-4ab0-8c70-59970dc01209-istio-token" pod "7758bd21-5416-4ab0-8c70-59970dc01209" (UID: "7758bd21-5416-4ab0-8c70-59970dc01209"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:34:02.654027 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:02.654011 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7758bd21-5416-4ab0-8c70-59970dc01209-kube-api-access-z7ms2" pod "7758bd21-5416-4ab0-8c70-59970dc01209" (UID: "7758bd21-5416-4ab0-8c70-59970dc01209"). InnerVolumeSpecName "kube-api-access-z7ms2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:34:02.666734 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:02.666705 4292 generic.go:356] "Generic (PLEG): container finished" podID="7758bd21-5416-4ab0-8c70-59970dc01209" containerID="3d02355f58c54ce2360837abf8ca61b881af6f3d03c70941b2712da765dc8999" exitCode=0 Oct 30 20:34:02.666849 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:02.666757 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl" Oct 30 20:34:02.666849 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:02.666785 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl" event={"ID":"7758bd21-5416-4ab0-8c70-59970dc01209","Type":"ContainerDied","Data":"3d02355f58c54ce2360837abf8ca61b881af6f3d03c70941b2712da765dc8999"} Oct 30 20:34:02.666849 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:02.666844 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl" event={"ID":"7758bd21-5416-4ab0-8c70-59970dc01209","Type":"ContainerDied","Data":"5e39370ef99b621ec9ec9bc6205fcea94745bfb4d2a6855d8d24d3cf735d3e37"} Oct 30 20:34:02.667006 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:02.666875 4292 scope.go:122] "RemoveContainer" containerID="3d02355f58c54ce2360837abf8ca61b881af6f3d03c70941b2712da765dc8999" Oct 30 20:34:02.668993 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:02.668969 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cafbb2-nhj6g" event={"ID":"a5379527-3946-433f-8abb-a0d43cc70cde","Type":"ContainerStarted","Data":"f8644de2cd0609206ecb3da5e1afa59900b8fb9413fe896e7becf025b6a442ee"} Oct 30 20:34:02.669135 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:02.669016 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cafbb2-nhj6g" event={"ID":"a5379527-3946-433f-8abb-a0d43cc70cde","Type":"ContainerStarted","Data":"ef7d03f72a0b181c4ce10a7cf6b396cb58049fd6b1c4ee1de968b5c9f5e6ca22"} Oct 30 20:34:02.698513 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:02.698257 4292 scope.go:122] "RemoveContainer" containerID="22d0158a2b2bf2aa55272736c8fc7d1c78739f5d782aa32f1def9a5c0a625dbf" Oct 30 20:34:02.713499 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:02.713465 4292 scope.go:122] "RemoveContainer" containerID="3d02355f58c54ce2360837abf8ca61b881af6f3d03c70941b2712da765dc8999" Oct 30 20:34:02.713878 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:34:02.713838 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d02355f58c54ce2360837abf8ca61b881af6f3d03c70941b2712da765dc8999\": container with ID starting with 3d02355f58c54ce2360837abf8ca61b881af6f3d03c70941b2712da765dc8999 not found: ID does not exist" containerID="3d02355f58c54ce2360837abf8ca61b881af6f3d03c70941b2712da765dc8999" Oct 30 20:34:02.713994 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:02.713876 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d02355f58c54ce2360837abf8ca61b881af6f3d03c70941b2712da765dc8999"} err="failed to get container status \"3d02355f58c54ce2360837abf8ca61b881af6f3d03c70941b2712da765dc8999\": rpc error: code = NotFound desc = could not find container \"3d02355f58c54ce2360837abf8ca61b881af6f3d03c70941b2712da765dc8999\": container with ID starting with 3d02355f58c54ce2360837abf8ca61b881af6f3d03c70941b2712da765dc8999 not found: ID does not exist" Oct 30 20:34:02.713994 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:02.713901 4292 scope.go:122] "RemoveContainer" containerID="22d0158a2b2bf2aa55272736c8fc7d1c78739f5d782aa32f1def9a5c0a625dbf" Oct 30 20:34:02.714317 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:34:02.714270 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22d0158a2b2bf2aa55272736c8fc7d1c78739f5d782aa32f1def9a5c0a625dbf\": container with ID starting with 22d0158a2b2bf2aa55272736c8fc7d1c78739f5d782aa32f1def9a5c0a625dbf not found: ID does not exist" containerID="22d0158a2b2bf2aa55272736c8fc7d1c78739f5d782aa32f1def9a5c0a625dbf" Oct 30 20:34:02.714450 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:02.714327 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22d0158a2b2bf2aa55272736c8fc7d1c78739f5d782aa32f1def9a5c0a625dbf"} err="failed to get container status \"22d0158a2b2bf2aa55272736c8fc7d1c78739f5d782aa32f1def9a5c0a625dbf\": rpc error: code = NotFound desc = could not find container \"22d0158a2b2bf2aa55272736c8fc7d1c78739f5d782aa32f1def9a5c0a625dbf\": container with ID starting with 22d0158a2b2bf2aa55272736c8fc7d1c78739f5d782aa32f1def9a5c0a625dbf not found: ID does not exist" Oct 30 20:34:02.753422 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:02.753394 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7758bd21-5416-4ab0-8c70-59970dc01209-workload-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:34:02.753422 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:02.753423 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7758bd21-5416-4ab0-8c70-59970dc01209-istio-token\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:34:02.753588 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:02.753450 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7758bd21-5416-4ab0-8c70-59970dc01209-istio-podinfo\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:34:02.753588 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:02.753562 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7758bd21-5416-4ab0-8c70-59970dc01209-istio-envoy\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:34:02.753673 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:02.753593 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7ms2\" (UniqueName: \"kubernetes.io/projected/7758bd21-5416-4ab0-8c70-59970dc01209-kube-api-access-z7ms2\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:34:02.864576 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:02.864516 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cafbb2-nhj6g" podStartSLOduration=1.8644962330000001 podStartE2EDuration="1.864496233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:34:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:34:02.863004269 +0000 UTC m=+4146.381319743" watchObservedRunningTime="2026-10-30 20:34:02.864496233 +0000 UTC m=+4146.382811690" Oct 30 20:34:02.979275 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:02.979241 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl"] Oct 30 20:34:02.984307 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:02.984278 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-742ff8-openshift-default-77b774b865-xf6fl"] Oct 30 20:34:03.676693 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:03.676656 4292 generic.go:356] "Generic (PLEG): container finished" podID="a5379527-3946-433f-8abb-a0d43cc70cde" containerID="f8644de2cd0609206ecb3da5e1afa59900b8fb9413fe896e7becf025b6a442ee" exitCode=0 Oct 30 20:34:03.677064 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:03.676733 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cafbb2-nhj6g" event={"ID":"a5379527-3946-433f-8abb-a0d43cc70cde","Type":"ContainerDied","Data":"f8644de2cd0609206ecb3da5e1afa59900b8fb9413fe896e7becf025b6a442ee"} Oct 30 20:34:04.364479 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:04.364455 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d" Oct 30 20:34:04.471441 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:04.471362 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/476ee650-d28b-4580-9eb4-a9f095df7822-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/476ee650-d28b-4580-9eb4-a9f095df7822-istio-podinfo\") pod \"476ee650-d28b-4580-9eb4-a9f095df7822\" (UID: \"476ee650-d28b-4580-9eb4-a9f095df7822\") " Oct 30 20:34:04.471441 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:04.471427 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/476ee650-d28b-4580-9eb4-a9f095df7822-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/476ee650-d28b-4580-9eb4-a9f095df7822-istio-envoy\") pod \"476ee650-d28b-4580-9eb4-a9f095df7822\" (UID: \"476ee650-d28b-4580-9eb4-a9f095df7822\") " Oct 30 20:34:04.471651 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:04.471472 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/476ee650-d28b-4580-9eb4-a9f095df7822-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/476ee650-d28b-4580-9eb4-a9f095df7822-workload-socket\") pod \"476ee650-d28b-4580-9eb4-a9f095df7822\" (UID: \"476ee650-d28b-4580-9eb4-a9f095df7822\") " Oct 30 20:34:04.471651 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:04.471495 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/476ee650-d28b-4580-9eb4-a9f095df7822-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/476ee650-d28b-4580-9eb4-a9f095df7822-workload-certs\") pod \"476ee650-d28b-4580-9eb4-a9f095df7822\" (UID: \"476ee650-d28b-4580-9eb4-a9f095df7822\") " Oct 30 20:34:04.471651 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:04.471522 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/476ee650-d28b-4580-9eb4-a9f095df7822-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/476ee650-d28b-4580-9eb4-a9f095df7822-credential-socket\") pod \"476ee650-d28b-4580-9eb4-a9f095df7822\" (UID: \"476ee650-d28b-4580-9eb4-a9f095df7822\") " Oct 30 20:34:04.471651 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:04.471563 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/476ee650-d28b-4580-9eb4-a9f095df7822-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/476ee650-d28b-4580-9eb4-a9f095df7822-istio-data\") pod \"476ee650-d28b-4580-9eb4-a9f095df7822\" (UID: \"476ee650-d28b-4580-9eb4-a9f095df7822\") " Oct 30 20:34:04.471651 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:04.471599 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/476ee650-d28b-4580-9eb4-a9f095df7822-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/476ee650-d28b-4580-9eb4-a9f095df7822-istiod-ca-cert\") pod \"476ee650-d28b-4580-9eb4-a9f095df7822\" (UID: \"476ee650-d28b-4580-9eb4-a9f095df7822\") " Oct 30 20:34:04.471906 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:04.471663 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/476ee650-d28b-4580-9eb4-a9f095df7822-istio-token\" (UniqueName: \"kubernetes.io/projected/476ee650-d28b-4580-9eb4-a9f095df7822-istio-token\") pod \"476ee650-d28b-4580-9eb4-a9f095df7822\" (UID: \"476ee650-d28b-4580-9eb4-a9f095df7822\") " Oct 30 20:34:04.471906 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:04.471694 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/476ee650-d28b-4580-9eb4-a9f095df7822-kube-api-access-xnhnd\" (UniqueName: \"kubernetes.io/projected/476ee650-d28b-4580-9eb4-a9f095df7822-kube-api-access-xnhnd\") pod \"476ee650-d28b-4580-9eb4-a9f095df7822\" (UID: \"476ee650-d28b-4580-9eb4-a9f095df7822\") " Oct 30 20:34:04.471906 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:04.471819 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/476ee650-d28b-4580-9eb4-a9f095df7822-workload-certs" pod "476ee650-d28b-4580-9eb4-a9f095df7822" (UID: "476ee650-d28b-4580-9eb4-a9f095df7822"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:34:04.471906 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:04.471891 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/476ee650-d28b-4580-9eb4-a9f095df7822-workload-socket" pod "476ee650-d28b-4580-9eb4-a9f095df7822" (UID: "476ee650-d28b-4580-9eb4-a9f095df7822"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:34:04.472086 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:04.471913 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/476ee650-d28b-4580-9eb4-a9f095df7822-credential-socket" pod "476ee650-d28b-4580-9eb4-a9f095df7822" (UID: "476ee650-d28b-4580-9eb4-a9f095df7822"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:34:04.472132 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:04.472110 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/476ee650-d28b-4580-9eb4-a9f095df7822-istiod-ca-cert" pod "476ee650-d28b-4580-9eb4-a9f095df7822" (UID: "476ee650-d28b-4580-9eb4-a9f095df7822"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:34:04.472263 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:04.472237 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/476ee650-d28b-4580-9eb4-a9f095df7822-istio-data" pod "476ee650-d28b-4580-9eb4-a9f095df7822" (UID: "476ee650-d28b-4580-9eb4-a9f095df7822"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:34:04.472312 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:04.472283 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/476ee650-d28b-4580-9eb4-a9f095df7822-workload-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:34:04.472312 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:04.472307 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/476ee650-d28b-4580-9eb4-a9f095df7822-workload-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:34:04.472406 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:04.472322 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/476ee650-d28b-4580-9eb4-a9f095df7822-credential-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:34:04.472406 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:04.472344 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/476ee650-d28b-4580-9eb4-a9f095df7822-istiod-ca-cert\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:34:04.474178 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:04.474154 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/476ee650-d28b-4580-9eb4-a9f095df7822-istio-envoy" pod "476ee650-d28b-4580-9eb4-a9f095df7822" (UID: "476ee650-d28b-4580-9eb4-a9f095df7822"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:34:04.474178 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:04.474168 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/476ee650-d28b-4580-9eb4-a9f095df7822-istio-podinfo" pod "476ee650-d28b-4580-9eb4-a9f095df7822" (UID: "476ee650-d28b-4580-9eb4-a9f095df7822"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:34:04.474331 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:04.474156 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/476ee650-d28b-4580-9eb4-a9f095df7822-kube-api-access-xnhnd" pod "476ee650-d28b-4580-9eb4-a9f095df7822" (UID: "476ee650-d28b-4580-9eb4-a9f095df7822"). InnerVolumeSpecName "kube-api-access-xnhnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:34:04.474331 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:04.474190 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/476ee650-d28b-4580-9eb4-a9f095df7822-istio-token" pod "476ee650-d28b-4580-9eb4-a9f095df7822" (UID: "476ee650-d28b-4580-9eb4-a9f095df7822"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:34:04.573469 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:04.573428 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/476ee650-d28b-4580-9eb4-a9f095df7822-istio-token\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:34:04.573469 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:04.573455 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnhnd\" (UniqueName: \"kubernetes.io/projected/476ee650-d28b-4580-9eb4-a9f095df7822-kube-api-access-xnhnd\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:34:04.573469 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:04.573467 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/476ee650-d28b-4580-9eb4-a9f095df7822-istio-podinfo\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:34:04.573469 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:04.573475 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/476ee650-d28b-4580-9eb4-a9f095df7822-istio-envoy\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:34:04.573837 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:04.573483 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/476ee650-d28b-4580-9eb4-a9f095df7822-istio-data\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:34:04.581972 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:04.581939 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7758bd21-5416-4ab0-8c70-59970dc01209" path="/var/lib/kubelet/pods/7758bd21-5416-4ab0-8c70-59970dc01209/volumes" Oct 30 20:34:04.684322 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:04.684287 4292 generic.go:356] "Generic (PLEG): container finished" podID="476ee650-d28b-4580-9eb4-a9f095df7822" containerID="45750ddbb4237d69d75e88edb1ec8d5d39c3fc2f221cdc744cabf0ad758416f6" exitCode=0 Oct 30 20:34:04.684770 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:04.684339 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d" Oct 30 20:34:04.684770 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:04.684381 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d" event={"ID":"476ee650-d28b-4580-9eb4-a9f095df7822","Type":"ContainerDied","Data":"45750ddbb4237d69d75e88edb1ec8d5d39c3fc2f221cdc744cabf0ad758416f6"} Oct 30 20:34:04.684770 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:04.684462 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d" event={"ID":"476ee650-d28b-4580-9eb4-a9f095df7822","Type":"ContainerDied","Data":"2bc3ed5657fa9701ba5bf52d7549ec20aec86a8209a450e62d5787525ad8a2bb"} Oct 30 20:34:04.684770 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:04.684486 4292 scope.go:122] "RemoveContainer" containerID="45750ddbb4237d69d75e88edb1ec8d5d39c3fc2f221cdc744cabf0ad758416f6" Oct 30 20:34:04.694856 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:04.694829 4292 scope.go:122] "RemoveContainer" containerID="dd97bc70b890574bee4c8fbf27e6c305188815e9e0c39c0e353c371201c010bc" Oct 30 20:34:04.703981 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:04.703950 4292 scope.go:122] "RemoveContainer" containerID="45750ddbb4237d69d75e88edb1ec8d5d39c3fc2f221cdc744cabf0ad758416f6" Oct 30 20:34:04.704253 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:34:04.704235 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45750ddbb4237d69d75e88edb1ec8d5d39c3fc2f221cdc744cabf0ad758416f6\": container with ID starting with 45750ddbb4237d69d75e88edb1ec8d5d39c3fc2f221cdc744cabf0ad758416f6 not found: ID does not exist" containerID="45750ddbb4237d69d75e88edb1ec8d5d39c3fc2f221cdc744cabf0ad758416f6" Oct 30 20:34:04.704388 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:04.704261 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45750ddbb4237d69d75e88edb1ec8d5d39c3fc2f221cdc744cabf0ad758416f6"} err="failed to get container status \"45750ddbb4237d69d75e88edb1ec8d5d39c3fc2f221cdc744cabf0ad758416f6\": rpc error: code = NotFound desc = could not find container \"45750ddbb4237d69d75e88edb1ec8d5d39c3fc2f221cdc744cabf0ad758416f6\": container with ID starting with 45750ddbb4237d69d75e88edb1ec8d5d39c3fc2f221cdc744cabf0ad758416f6 not found: ID does not exist" Oct 30 20:34:04.704388 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:04.704283 4292 scope.go:122] "RemoveContainer" containerID="dd97bc70b890574bee4c8fbf27e6c305188815e9e0c39c0e353c371201c010bc" Oct 30 20:34:04.704572 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:34:04.704553 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd97bc70b890574bee4c8fbf27e6c305188815e9e0c39c0e353c371201c010bc\": container with ID starting with dd97bc70b890574bee4c8fbf27e6c305188815e9e0c39c0e353c371201c010bc not found: ID does not exist" containerID="dd97bc70b890574bee4c8fbf27e6c305188815e9e0c39c0e353c371201c010bc" Oct 30 20:34:04.704630 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:04.704581 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd97bc70b890574bee4c8fbf27e6c305188815e9e0c39c0e353c371201c010bc"} err="failed to get container status \"dd97bc70b890574bee4c8fbf27e6c305188815e9e0c39c0e353c371201c010bc\": rpc error: code = NotFound desc = could not find container \"dd97bc70b890574bee4c8fbf27e6c305188815e9e0c39c0e353c371201c010bc\": container with ID starting with dd97bc70b890574bee4c8fbf27e6c305188815e9e0c39c0e353c371201c010bc not found: ID does not exist" Oct 30 20:34:04.814675 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:04.814651 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cafbb2-nhj6g" Oct 30 20:34:04.854560 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:04.854530 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d"] Oct 30 20:34:04.905196 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:04.905168 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-8d612c-openshift-default-84d87678c7-gpp9d"] Oct 30 20:34:04.977869 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:04.977834 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5379527-3946-433f-8abb-a0d43cc70cde-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5379527-3946-433f-8abb-a0d43cc70cde-maas-api-openshift-ca\") pod \"a5379527-3946-433f-8abb-a0d43cc70cde\" (UID: \"a5379527-3946-433f-8abb-a0d43cc70cde\") " Oct 30 20:34:04.978056 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:04.977913 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5379527-3946-433f-8abb-a0d43cc70cde-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5379527-3946-433f-8abb-a0d43cc70cde-maas-api-xks-ca\") pod \"a5379527-3946-433f-8abb-a0d43cc70cde\" (UID: \"a5379527-3946-433f-8abb-a0d43cc70cde\") " Oct 30 20:34:04.978056 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:04.978040 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5379527-3946-433f-8abb-a0d43cc70cde-kube-api-access-dp8vl\" (UniqueName: \"kubernetes.io/projected/a5379527-3946-433f-8abb-a0d43cc70cde-kube-api-access-dp8vl\") pod \"a5379527-3946-433f-8abb-a0d43cc70cde\" (UID: \"a5379527-3946-433f-8abb-a0d43cc70cde\") " Oct 30 20:34:04.978663 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:04.978633 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5379527-3946-433f-8abb-a0d43cc70cde-maas-api-openshift-ca" pod "a5379527-3946-433f-8abb-a0d43cc70cde" (UID: "a5379527-3946-433f-8abb-a0d43cc70cde"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:34:04.980676 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:04.980645 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5379527-3946-433f-8abb-a0d43cc70cde-maas-api-xks-ca" pod "a5379527-3946-433f-8abb-a0d43cc70cde" (UID: "a5379527-3946-433f-8abb-a0d43cc70cde"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:34:04.980793 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:04.980749 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5379527-3946-433f-8abb-a0d43cc70cde-kube-api-access-dp8vl" pod "a5379527-3946-433f-8abb-a0d43cc70cde" (UID: "a5379527-3946-433f-8abb-a0d43cc70cde"). InnerVolumeSpecName "kube-api-access-dp8vl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:34:05.078959 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:05.078931 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5379527-3946-433f-8abb-a0d43cc70cde-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:34:05.078959 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:05.078958 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5379527-3946-433f-8abb-a0d43cc70cde-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:34:05.079131 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:05.078969 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dp8vl\" (UniqueName: \"kubernetes.io/projected/a5379527-3946-433f-8abb-a0d43cc70cde-kube-api-access-dp8vl\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:34:05.692288 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:05.692252 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cafbb2-nhj6g" event={"ID":"a5379527-3946-433f-8abb-a0d43cc70cde","Type":"ContainerDied","Data":"ef7d03f72a0b181c4ce10a7cf6b396cb58049fd6b1c4ee1de968b5c9f5e6ca22"} Oct 30 20:34:05.692288 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:05.692283 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cafbb2-nhj6g" Oct 30 20:34:05.692763 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:05.692308 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef7d03f72a0b181c4ce10a7cf6b396cb58049fd6b1c4ee1de968b5c9f5e6ca22" Oct 30 20:34:06.582371 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:06.582331 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="476ee650-d28b-4580-9eb4-a9f095df7822" path="/var/lib/kubelet/pods/476ee650-d28b-4580-9eb4-a9f095df7822/volumes" Oct 30 20:34:09.670399 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:09.670361 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-b91df1-765b88fb56-l2xpb"] Oct 30 20:34:09.779632 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:09.779599 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b91df1-5c7bbf65f-vvg2l"] Oct 30 20:34:12.058966 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:12.058933 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b52b4147-gmggr"] Oct 30 20:34:12.080030 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:12.080002 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b52b4147-gmggr"] Oct 30 20:34:12.582524 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:12.582494 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35366a1c-b8dc-40e6-b592-296d351702b5" path="/var/lib/kubelet/pods/35366a1c-b8dc-40e6-b592-296d351702b5/volumes" Oct 30 20:34:13.220004 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:34:13.219969 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:34:13.351512 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:13.351478 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0e72bd-h8vxx"] Oct 30 20:34:13.351932 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:13.351920 4292 cpu_manager.go:414] "containerMap: removing container" podUID="7758bd21-5416-4ab0-8c70-59970dc01209" containerName="istio-proxy" Oct 30 20:34:13.351980 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:13.351933 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7758bd21-5416-4ab0-8c70-59970dc01209" containerName="istio-proxy" Oct 30 20:34:13.351980 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:13.351946 4292 cpu_manager.go:414] "containerMap: removing container" podUID="7758bd21-5416-4ab0-8c70-59970dc01209" containerName="istio-proxy" Oct 30 20:34:13.351980 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:13.351951 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7758bd21-5416-4ab0-8c70-59970dc01209" containerName="istio-proxy" Oct 30 20:34:13.351980 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:13.351959 4292 cpu_manager.go:414] "containerMap: removing container" podUID="476ee650-d28b-4580-9eb4-a9f095df7822" containerName="istio-proxy" Oct 30 20:34:13.351980 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:13.351964 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="476ee650-d28b-4580-9eb4-a9f095df7822" containerName="istio-proxy" Oct 30 20:34:13.351980 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:13.351971 4292 cpu_manager.go:414] "containerMap: removing container" podUID="476ee650-d28b-4580-9eb4-a9f095df7822" containerName="istio-proxy" Oct 30 20:34:13.351980 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:13.351975 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="476ee650-d28b-4580-9eb4-a9f095df7822" containerName="istio-proxy" Oct 30 20:34:13.352223 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:13.351992 4292 cpu_manager.go:414] "containerMap: removing container" podUID="a5379527-3946-433f-8abb-a0d43cc70cde" containerName="revoke-keys" Oct 30 20:34:13.352223 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:13.351997 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5379527-3946-433f-8abb-a0d43cc70cde" containerName="revoke-keys" Oct 30 20:34:13.352223 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:13.352066 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="7758bd21-5416-4ab0-8c70-59970dc01209" containerName="istio-proxy" Oct 30 20:34:13.352223 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:13.352075 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="7758bd21-5416-4ab0-8c70-59970dc01209" containerName="istio-proxy" Oct 30 20:34:13.352223 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:13.352081 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="476ee650-d28b-4580-9eb4-a9f095df7822" containerName="istio-proxy" Oct 30 20:34:13.352223 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:13.352087 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="476ee650-d28b-4580-9eb4-a9f095df7822" containerName="istio-proxy" Oct 30 20:34:13.352223 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:13.352096 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5379527-3946-433f-8abb-a0d43cc70cde" containerName="revoke-keys" Oct 30 20:34:13.355389 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:13.355373 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0e72bd-h8vxx" Oct 30 20:34:13.462135 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:13.462107 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:34:13.462135 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:13.462130 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\"" Oct 30 20:34:13.464760 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:13.464741 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0e72bd-h8vxx"] Oct 30 20:34:13.477886 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:13.477827 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f3a292e-96b6-411b-9503-1b5a74c7f5ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-0e72bd-h8vxx\" (UID: \"4f3a292e-96b6-411b-9503-1b5a74c7f5ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0e72bd-h8vxx" Oct 30 20:34:13.477886 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:13.477867 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f3a292e-96b6-411b-9503-1b5a74c7f5ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-0e72bd-h8vxx\" (UID: \"4f3a292e-96b6-411b-9503-1b5a74c7f5ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0e72bd-h8vxx" Oct 30 20:34:13.478037 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:13.477995 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vf5k\" (UniqueName: \"kubernetes.io/projected/4f3a292e-96b6-411b-9503-1b5a74c7f5ab-kube-api-access-8vf5k\") pod \"maas-api-revoke-keys-e2e-extmod-a-0e72bd-h8vxx\" (UID: \"4f3a292e-96b6-411b-9503-1b5a74c7f5ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0e72bd-h8vxx" Oct 30 20:34:13.578820 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:13.578789 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f3a292e-96b6-411b-9503-1b5a74c7f5ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-0e72bd-h8vxx\" (UID: \"4f3a292e-96b6-411b-9503-1b5a74c7f5ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0e72bd-h8vxx" Oct 30 20:34:13.579013 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:13.578848 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f3a292e-96b6-411b-9503-1b5a74c7f5ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-0e72bd-h8vxx\" (UID: \"4f3a292e-96b6-411b-9503-1b5a74c7f5ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0e72bd-h8vxx" Oct 30 20:34:13.579013 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:13.578948 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vf5k\" (UniqueName: \"kubernetes.io/projected/4f3a292e-96b6-411b-9503-1b5a74c7f5ab-kube-api-access-8vf5k\") pod \"maas-api-revoke-keys-e2e-extmod-a-0e72bd-h8vxx\" (UID: \"4f3a292e-96b6-411b-9503-1b5a74c7f5ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0e72bd-h8vxx" Oct 30 20:34:13.579743 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:13.579713 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f3a292e-96b6-411b-9503-1b5a74c7f5ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-0e72bd-h8vxx\" (UID: \"4f3a292e-96b6-411b-9503-1b5a74c7f5ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0e72bd-h8vxx" Oct 30 20:34:13.581466 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:13.581441 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f3a292e-96b6-411b-9503-1b5a74c7f5ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-0e72bd-h8vxx\" (UID: \"4f3a292e-96b6-411b-9503-1b5a74c7f5ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0e72bd-h8vxx" Oct 30 20:34:13.643313 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:13.643284 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vf5k\" (UniqueName: \"kubernetes.io/projected/4f3a292e-96b6-411b-9503-1b5a74c7f5ab-kube-api-access-8vf5k\") pod \"maas-api-revoke-keys-e2e-extmod-a-0e72bd-h8vxx\" (UID: \"4f3a292e-96b6-411b-9503-1b5a74c7f5ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0e72bd-h8vxx" Oct 30 20:34:13.668264 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:13.668242 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0e72bd-h8vxx" Oct 30 20:34:14.113654 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:14.113622 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0e72bd-h8vxx"] Oct 30 20:34:14.115227 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:34:14.115203 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f3a292e_96b6_411b_9503_1b5a74c7f5ab.slice/crio-4fee27fc7d4c8443fe8c3345421c1bdfe581e73a73f5c8459e52f332c8a5d646 WatchSource:0}: Error finding container 4fee27fc7d4c8443fe8c3345421c1bdfe581e73a73f5c8459e52f332c8a5d646: Status 404 returned error can't find the container with id 4fee27fc7d4c8443fe8c3345421c1bdfe581e73a73f5c8459e52f332c8a5d646 Oct 30 20:34:14.671159 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:14.671119 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b91df1-765b88fb56-l2xpb" podUID="335d81c9-5d05-42b9-8d9b-90974d402605" containerName="payload-processing" containerID="cri-o://14a296991022fb52878c2eea4e125e7e0b21f241ea5827fd9d3578376bcbdbc1" gracePeriod=25 Oct 30 20:34:14.780482 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:14.780412 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b91df1-5c7bbf65f-vvg2l" podUID="b07a622c-18d8-4351-a871-6b019ba2a2a4" containerName="payload-pre-processing" containerID="cri-o://f0e544a3c259b6d8e9fae717944a1827849036674765c5a974b8ea30674d13e0" gracePeriod=25 Oct 30 20:34:14.791197 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:14.791169 4292 generic.go:356] "Generic (PLEG): container finished" podID="335d81c9-5d05-42b9-8d9b-90974d402605" containerID="14a296991022fb52878c2eea4e125e7e0b21f241ea5827fd9d3578376bcbdbc1" exitCode=0 Oct 30 20:34:14.791302 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:14.791239 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b91df1-765b88fb56-l2xpb" event={"ID":"335d81c9-5d05-42b9-8d9b-90974d402605","Type":"ContainerDied","Data":"14a296991022fb52878c2eea4e125e7e0b21f241ea5827fd9d3578376bcbdbc1"} Oct 30 20:34:14.792860 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:14.792839 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0e72bd-h8vxx" event={"ID":"4f3a292e-96b6-411b-9503-1b5a74c7f5ab","Type":"ContainerStarted","Data":"655400de033270da71a67e78a2aeaa16d9b012b11cda17d7609bad74f30305ed"} Oct 30 20:34:14.792946 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:14.792877 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0e72bd-h8vxx" event={"ID":"4f3a292e-96b6-411b-9503-1b5a74c7f5ab","Type":"ContainerStarted","Data":"4fee27fc7d4c8443fe8c3345421c1bdfe581e73a73f5c8459e52f332c8a5d646"} Oct 30 20:34:14.912254 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:14.912202 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0e72bd-h8vxx" podStartSLOduration=1.912190303 podStartE2EDuration="1.912190303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:34:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:34:14.91169824 +0000 UTC m=+4158.430013698" watchObservedRunningTime="2026-10-30 20:34:14.912190303 +0000 UTC m=+4158.430505776" Oct 30 20:34:14.939697 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:14.939666 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b91df1-765b88fb56-l2xpb" Oct 30 20:34:14.996091 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:14.996058 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/335d81c9-5d05-42b9-8d9b-90974d402605-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/335d81c9-5d05-42b9-8d9b-90974d402605-plugins-config-volume\") pod \"335d81c9-5d05-42b9-8d9b-90974d402605\" (UID: \"335d81c9-5d05-42b9-8d9b-90974d402605\") " Oct 30 20:34:14.996258 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:14.996169 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/335d81c9-5d05-42b9-8d9b-90974d402605-kube-api-access-d2x75\" (UniqueName: \"kubernetes.io/projected/335d81c9-5d05-42b9-8d9b-90974d402605-kube-api-access-d2x75\") pod \"335d81c9-5d05-42b9-8d9b-90974d402605\" (UID: \"335d81c9-5d05-42b9-8d9b-90974d402605\") " Oct 30 20:34:14.997113 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:14.997081 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/335d81c9-5d05-42b9-8d9b-90974d402605-plugins-config-volume" pod "335d81c9-5d05-42b9-8d9b-90974d402605" (UID: "335d81c9-5d05-42b9-8d9b-90974d402605"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:34:14.999941 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:14.999913 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/335d81c9-5d05-42b9-8d9b-90974d402605-kube-api-access-d2x75" pod "335d81c9-5d05-42b9-8d9b-90974d402605" (UID: "335d81c9-5d05-42b9-8d9b-90974d402605"). InnerVolumeSpecName "kube-api-access-d2x75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:34:15.029982 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:15.029947 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b91df1-5c7bbf65f-vvg2l" Oct 30 20:34:15.097670 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:15.097641 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b07a622c-18d8-4351-a871-6b019ba2a2a4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b07a622c-18d8-4351-a871-6b019ba2a2a4-plugins-config-volume\") pod \"b07a622c-18d8-4351-a871-6b019ba2a2a4\" (UID: \"b07a622c-18d8-4351-a871-6b019ba2a2a4\") " Oct 30 20:34:15.097831 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:15.097706 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b07a622c-18d8-4351-a871-6b019ba2a2a4-kube-api-access-c675v\" (UniqueName: \"kubernetes.io/projected/b07a622c-18d8-4351-a871-6b019ba2a2a4-kube-api-access-c675v\") pod \"b07a622c-18d8-4351-a871-6b019ba2a2a4\" (UID: \"b07a622c-18d8-4351-a871-6b019ba2a2a4\") " Oct 30 20:34:15.098001 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:15.097981 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/335d81c9-5d05-42b9-8d9b-90974d402605-plugins-config-volume\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:34:15.098001 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:15.097997 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2x75\" (UniqueName: \"kubernetes.io/projected/335d81c9-5d05-42b9-8d9b-90974d402605-kube-api-access-d2x75\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:34:15.098159 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:15.098044 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b07a622c-18d8-4351-a871-6b019ba2a2a4-plugins-config-volume" pod "b07a622c-18d8-4351-a871-6b019ba2a2a4" (UID: "b07a622c-18d8-4351-a871-6b019ba2a2a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:34:15.099773 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:15.099750 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b07a622c-18d8-4351-a871-6b019ba2a2a4-kube-api-access-c675v" pod "b07a622c-18d8-4351-a871-6b019ba2a2a4" (UID: "b07a622c-18d8-4351-a871-6b019ba2a2a4"). InnerVolumeSpecName "kube-api-access-c675v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:34:15.198952 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:15.198881 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b07a622c-18d8-4351-a871-6b019ba2a2a4-plugins-config-volume\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:34:15.198952 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:15.198908 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c675v\" (UniqueName: \"kubernetes.io/projected/b07a622c-18d8-4351-a871-6b019ba2a2a4-kube-api-access-c675v\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:34:15.799585 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:15.799556 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b91df1-765b88fb56-l2xpb" Oct 30 20:34:15.799585 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:15.799571 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b91df1-765b88fb56-l2xpb" event={"ID":"335d81c9-5d05-42b9-8d9b-90974d402605","Type":"ContainerDied","Data":"bcc298ab18fbf32e7e964f9a6f0fe7889007b332353ef8a0635175edc9f73781"} Oct 30 20:34:15.800069 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:15.799636 4292 scope.go:122] "RemoveContainer" containerID="14a296991022fb52878c2eea4e125e7e0b21f241ea5827fd9d3578376bcbdbc1" Oct 30 20:34:15.801262 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:15.801237 4292 generic.go:356] "Generic (PLEG): container finished" podID="4f3a292e-96b6-411b-9503-1b5a74c7f5ab" containerID="655400de033270da71a67e78a2aeaa16d9b012b11cda17d7609bad74f30305ed" exitCode=0 Oct 30 20:34:15.801493 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:15.801336 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0e72bd-h8vxx" event={"ID":"4f3a292e-96b6-411b-9503-1b5a74c7f5ab","Type":"ContainerDied","Data":"655400de033270da71a67e78a2aeaa16d9b012b11cda17d7609bad74f30305ed"} Oct 30 20:34:15.802696 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:15.802668 4292 generic.go:356] "Generic (PLEG): container finished" podID="b07a622c-18d8-4351-a871-6b019ba2a2a4" containerID="f0e544a3c259b6d8e9fae717944a1827849036674765c5a974b8ea30674d13e0" exitCode=0 Oct 30 20:34:15.802800 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:15.802718 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b91df1-5c7bbf65f-vvg2l" Oct 30 20:34:15.802874 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:15.802719 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b91df1-5c7bbf65f-vvg2l" event={"ID":"b07a622c-18d8-4351-a871-6b019ba2a2a4","Type":"ContainerDied","Data":"f0e544a3c259b6d8e9fae717944a1827849036674765c5a974b8ea30674d13e0"} Oct 30 20:34:15.802874 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:15.802846 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b91df1-5c7bbf65f-vvg2l" event={"ID":"b07a622c-18d8-4351-a871-6b019ba2a2a4","Type":"ContainerDied","Data":"ced609a98cd9adf084a5f570f59c4e927914cc0565935567967f6ce5bad30f16"} Oct 30 20:34:15.810081 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:15.810063 4292 scope.go:122] "RemoveContainer" containerID="f0e544a3c259b6d8e9fae717944a1827849036674765c5a974b8ea30674d13e0" Oct 30 20:34:15.818220 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:15.818204 4292 scope.go:122] "RemoveContainer" containerID="f0e544a3c259b6d8e9fae717944a1827849036674765c5a974b8ea30674d13e0" Oct 30 20:34:15.818519 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:34:15.818495 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0e544a3c259b6d8e9fae717944a1827849036674765c5a974b8ea30674d13e0\": container with ID starting with f0e544a3c259b6d8e9fae717944a1827849036674765c5a974b8ea30674d13e0 not found: ID does not exist" containerID="f0e544a3c259b6d8e9fae717944a1827849036674765c5a974b8ea30674d13e0" Oct 30 20:34:15.818619 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:15.818525 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0e544a3c259b6d8e9fae717944a1827849036674765c5a974b8ea30674d13e0"} err="failed to get container status \"f0e544a3c259b6d8e9fae717944a1827849036674765c5a974b8ea30674d13e0\": rpc error: code = NotFound desc = could not find container \"f0e544a3c259b6d8e9fae717944a1827849036674765c5a974b8ea30674d13e0\": container with ID starting with f0e544a3c259b6d8e9fae717944a1827849036674765c5a974b8ea30674d13e0 not found: ID does not exist" Oct 30 20:34:16.087444 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:16.087412 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-b91df1-765b88fb56-l2xpb"] Oct 30 20:34:16.091510 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:16.091477 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-b91df1-765b88fb56-l2xpb"] Oct 30 20:34:16.204042 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:16.204009 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b91df1-5c7bbf65f-vvg2l"] Oct 30 20:34:16.208798 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:16.208768 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b91df1-5c7bbf65f-vvg2l"] Oct 30 20:34:16.582910 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:16.582883 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="335d81c9-5d05-42b9-8d9b-90974d402605" path="/var/lib/kubelet/pods/335d81c9-5d05-42b9-8d9b-90974d402605/volumes" Oct 30 20:34:16.583273 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:16.583260 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b07a622c-18d8-4351-a871-6b019ba2a2a4" path="/var/lib/kubelet/pods/b07a622c-18d8-4351-a871-6b019ba2a2a4/volumes" Oct 30 20:34:16.939110 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:16.939088 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0e72bd-h8vxx" Oct 30 20:34:17.017735 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:17.017703 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f3a292e-96b6-411b-9503-1b5a74c7f5ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f3a292e-96b6-411b-9503-1b5a74c7f5ab-maas-api-openshift-ca\") pod \"4f3a292e-96b6-411b-9503-1b5a74c7f5ab\" (UID: \"4f3a292e-96b6-411b-9503-1b5a74c7f5ab\") " Oct 30 20:34:17.017882 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:17.017752 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f3a292e-96b6-411b-9503-1b5a74c7f5ab-kube-api-access-8vf5k\" (UniqueName: \"kubernetes.io/projected/4f3a292e-96b6-411b-9503-1b5a74c7f5ab-kube-api-access-8vf5k\") pod \"4f3a292e-96b6-411b-9503-1b5a74c7f5ab\" (UID: \"4f3a292e-96b6-411b-9503-1b5a74c7f5ab\") " Oct 30 20:34:17.017882 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:17.017793 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f3a292e-96b6-411b-9503-1b5a74c7f5ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f3a292e-96b6-411b-9503-1b5a74c7f5ab-maas-api-xks-ca\") pod \"4f3a292e-96b6-411b-9503-1b5a74c7f5ab\" (UID: \"4f3a292e-96b6-411b-9503-1b5a74c7f5ab\") " Oct 30 20:34:17.018060 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:17.018040 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f3a292e-96b6-411b-9503-1b5a74c7f5ab-maas-api-openshift-ca" pod "4f3a292e-96b6-411b-9503-1b5a74c7f5ab" (UID: "4f3a292e-96b6-411b-9503-1b5a74c7f5ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:34:17.018299 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:17.018276 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f3a292e-96b6-411b-9503-1b5a74c7f5ab-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:34:17.019882 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:17.019855 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f3a292e-96b6-411b-9503-1b5a74c7f5ab-maas-api-xks-ca" pod "4f3a292e-96b6-411b-9503-1b5a74c7f5ab" (UID: "4f3a292e-96b6-411b-9503-1b5a74c7f5ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:34:17.019959 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:17.019883 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f3a292e-96b6-411b-9503-1b5a74c7f5ab-kube-api-access-8vf5k" pod "4f3a292e-96b6-411b-9503-1b5a74c7f5ab" (UID: "4f3a292e-96b6-411b-9503-1b5a74c7f5ab"). InnerVolumeSpecName "kube-api-access-8vf5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:34:17.119809 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:17.119734 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vf5k\" (UniqueName: \"kubernetes.io/projected/4f3a292e-96b6-411b-9503-1b5a74c7f5ab-kube-api-access-8vf5k\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:34:17.119809 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:17.119759 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f3a292e-96b6-411b-9503-1b5a74c7f5ab-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:34:17.817644 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:17.817610 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0e72bd-h8vxx" Oct 30 20:34:17.817644 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:17.817632 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0e72bd-h8vxx" event={"ID":"4f3a292e-96b6-411b-9503-1b5a74c7f5ab","Type":"ContainerDied","Data":"4fee27fc7d4c8443fe8c3345421c1bdfe581e73a73f5c8459e52f332c8a5d646"} Oct 30 20:34:17.817859 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:17.817679 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4fee27fc7d4c8443fe8c3345421c1bdfe581e73a73f5c8459e52f332c8a5d646" Oct 30 20:34:37.753854 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:37.753816 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-fe49e7-cvz2l"] Oct 30 20:34:37.754385 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:37.754289 4292 cpu_manager.go:414] "containerMap: removing container" podUID="4f3a292e-96b6-411b-9503-1b5a74c7f5ab" containerName="revoke-keys" Oct 30 20:34:37.754385 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:37.754303 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f3a292e-96b6-411b-9503-1b5a74c7f5ab" containerName="revoke-keys" Oct 30 20:34:37.754385 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:37.754324 4292 cpu_manager.go:414] "containerMap: removing container" podUID="b07a622c-18d8-4351-a871-6b019ba2a2a4" containerName="payload-pre-processing" Oct 30 20:34:37.754385 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:37.754329 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b07a622c-18d8-4351-a871-6b019ba2a2a4" containerName="payload-pre-processing" Oct 30 20:34:37.754385 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:37.754362 4292 cpu_manager.go:414] "containerMap: removing container" podUID="335d81c9-5d05-42b9-8d9b-90974d402605" containerName="payload-processing" Oct 30 20:34:37.754385 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:37.754370 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="335d81c9-5d05-42b9-8d9b-90974d402605" containerName="payload-processing" Oct 30 20:34:37.754601 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:37.754473 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f3a292e-96b6-411b-9503-1b5a74c7f5ab" containerName="revoke-keys" Oct 30 20:34:37.754601 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:37.754486 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="b07a622c-18d8-4351-a871-6b019ba2a2a4" containerName="payload-pre-processing" Oct 30 20:34:37.754601 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:37.754499 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="335d81c9-5d05-42b9-8d9b-90974d402605" containerName="payload-processing" Oct 30 20:34:37.757611 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:37.757588 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-fe49e7-cvz2l" Oct 30 20:34:37.832496 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:37.832458 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:34:37.835123 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:37.835091 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db721103-aee1-434a-be3d-ef0d13a54b4d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-fe49e7-cvz2l\" (UID: \"db721103-aee1-434a-be3d-ef0d13a54b4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-fe49e7-cvz2l" Oct 30 20:34:37.835298 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:37.835146 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db721103-aee1-434a-be3d-ef0d13a54b4d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-fe49e7-cvz2l\" (UID: \"db721103-aee1-434a-be3d-ef0d13a54b4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-fe49e7-cvz2l" Oct 30 20:34:37.835298 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:37.835248 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqmxs\" (UniqueName: \"kubernetes.io/projected/db721103-aee1-434a-be3d-ef0d13a54b4d-kube-api-access-lqmxs\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-fe49e7-cvz2l\" (UID: \"db721103-aee1-434a-be3d-ef0d13a54b4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-fe49e7-cvz2l" Oct 30 20:34:37.862263 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:37.862233 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\"" Oct 30 20:34:37.865475 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:37.865448 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-fe49e7-cvz2l"] Oct 30 20:34:37.936706 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:37.936674 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db721103-aee1-434a-be3d-ef0d13a54b4d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-fe49e7-cvz2l\" (UID: \"db721103-aee1-434a-be3d-ef0d13a54b4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-fe49e7-cvz2l" Oct 30 20:34:37.936966 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:37.936725 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqmxs\" (UniqueName: \"kubernetes.io/projected/db721103-aee1-434a-be3d-ef0d13a54b4d-kube-api-access-lqmxs\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-fe49e7-cvz2l\" (UID: \"db721103-aee1-434a-be3d-ef0d13a54b4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-fe49e7-cvz2l" Oct 30 20:34:37.936966 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:37.936866 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db721103-aee1-434a-be3d-ef0d13a54b4d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-fe49e7-cvz2l\" (UID: \"db721103-aee1-434a-be3d-ef0d13a54b4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-fe49e7-cvz2l" Oct 30 20:34:37.937568 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:37.937543 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db721103-aee1-434a-be3d-ef0d13a54b4d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-fe49e7-cvz2l\" (UID: \"db721103-aee1-434a-be3d-ef0d13a54b4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-fe49e7-cvz2l" Oct 30 20:34:37.939131 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:37.939107 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db721103-aee1-434a-be3d-ef0d13a54b4d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-fe49e7-cvz2l\" (UID: \"db721103-aee1-434a-be3d-ef0d13a54b4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-fe49e7-cvz2l" Oct 30 20:34:38.000227 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:38.000194 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqmxs\" (UniqueName: \"kubernetes.io/projected/db721103-aee1-434a-be3d-ef0d13a54b4d-kube-api-access-lqmxs\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-fe49e7-cvz2l\" (UID: \"db721103-aee1-434a-be3d-ef0d13a54b4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-fe49e7-cvz2l" Oct 30 20:34:38.071138 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:38.071105 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-fe49e7-cvz2l" Oct 30 20:34:38.458783 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:38.458758 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-fe49e7-cvz2l"] Oct 30 20:34:38.460865 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:34:38.460835 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb721103_aee1_434a_be3d_ef0d13a54b4d.slice/crio-5fd7671b332eccf8b2999104569fd648045f321b7e30511b19bcffc3b86ac0ba WatchSource:0}: Error finding container 5fd7671b332eccf8b2999104569fd648045f321b7e30511b19bcffc3b86ac0ba: Status 404 returned error can't find the container with id 5fd7671b332eccf8b2999104569fd648045f321b7e30511b19bcffc3b86ac0ba Oct 30 20:34:38.939524 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:38.939487 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-fe49e7-cvz2l" event={"ID":"db721103-aee1-434a-be3d-ef0d13a54b4d","Type":"ContainerStarted","Data":"74db238b53b85c63a54657b90aea43b97008b23dd107d72774f4a659ed4bf226"} Oct 30 20:34:38.940056 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:38.939545 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-fe49e7-cvz2l" event={"ID":"db721103-aee1-434a-be3d-ef0d13a54b4d","Type":"ContainerStarted","Data":"5fd7671b332eccf8b2999104569fd648045f321b7e30511b19bcffc3b86ac0ba"} Oct 30 20:34:39.060305 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:39.060257 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-fe49e7-cvz2l" podStartSLOduration=2.060244441 podStartE2EDuration="2.060244441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:34:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:34:39.05820321 +0000 UTC m=+4182.576518685" watchObservedRunningTime="2026-10-30 20:34:39.060244441 +0000 UTC m=+4182.578559939" Oct 30 20:34:39.948666 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:39.948628 4292 generic.go:356] "Generic (PLEG): container finished" podID="db721103-aee1-434a-be3d-ef0d13a54b4d" containerID="74db238b53b85c63a54657b90aea43b97008b23dd107d72774f4a659ed4bf226" exitCode=0 Oct 30 20:34:39.949064 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:39.948713 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-fe49e7-cvz2l" event={"ID":"db721103-aee1-434a-be3d-ef0d13a54b4d","Type":"ContainerDied","Data":"74db238b53b85c63a54657b90aea43b97008b23dd107d72774f4a659ed4bf226"} Oct 30 20:34:41.094721 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:41.094691 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-fe49e7-cvz2l" Oct 30 20:34:41.175082 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:41.175058 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db721103-aee1-434a-be3d-ef0d13a54b4d-kube-api-access-lqmxs\" (UniqueName: \"kubernetes.io/projected/db721103-aee1-434a-be3d-ef0d13a54b4d-kube-api-access-lqmxs\") pod \"db721103-aee1-434a-be3d-ef0d13a54b4d\" (UID: \"db721103-aee1-434a-be3d-ef0d13a54b4d\") " Oct 30 20:34:41.175273 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:41.175125 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db721103-aee1-434a-be3d-ef0d13a54b4d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db721103-aee1-434a-be3d-ef0d13a54b4d-maas-api-openshift-ca\") pod \"db721103-aee1-434a-be3d-ef0d13a54b4d\" (UID: \"db721103-aee1-434a-be3d-ef0d13a54b4d\") " Oct 30 20:34:41.175273 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:41.175170 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db721103-aee1-434a-be3d-ef0d13a54b4d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db721103-aee1-434a-be3d-ef0d13a54b4d-maas-api-xks-ca\") pod \"db721103-aee1-434a-be3d-ef0d13a54b4d\" (UID: \"db721103-aee1-434a-be3d-ef0d13a54b4d\") " Oct 30 20:34:41.175510 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:41.175483 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db721103-aee1-434a-be3d-ef0d13a54b4d-maas-api-openshift-ca" pod "db721103-aee1-434a-be3d-ef0d13a54b4d" (UID: "db721103-aee1-434a-be3d-ef0d13a54b4d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:34:41.177173 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:41.177145 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db721103-aee1-434a-be3d-ef0d13a54b4d-maas-api-xks-ca" pod "db721103-aee1-434a-be3d-ef0d13a54b4d" (UID: "db721103-aee1-434a-be3d-ef0d13a54b4d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:34:41.177283 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:41.177242 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db721103-aee1-434a-be3d-ef0d13a54b4d-kube-api-access-lqmxs" pod "db721103-aee1-434a-be3d-ef0d13a54b4d" (UID: "db721103-aee1-434a-be3d-ef0d13a54b4d"). InnerVolumeSpecName "kube-api-access-lqmxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:34:41.276314 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:41.276284 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db721103-aee1-434a-be3d-ef0d13a54b4d-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:34:41.276314 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:41.276311 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db721103-aee1-434a-be3d-ef0d13a54b4d-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:34:41.276314 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:41.276321 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqmxs\" (UniqueName: \"kubernetes.io/projected/db721103-aee1-434a-be3d-ef0d13a54b4d-kube-api-access-lqmxs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:34:41.500554 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:41.500515 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cafbb2-nhj6g"] Oct 30 20:34:41.504244 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:41.504222 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cafbb2-nhj6g"] Oct 30 20:34:41.960180 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:41.960148 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-fe49e7-cvz2l" event={"ID":"db721103-aee1-434a-be3d-ef0d13a54b4d","Type":"ContainerDied","Data":"5fd7671b332eccf8b2999104569fd648045f321b7e30511b19bcffc3b86ac0ba"} Oct 30 20:34:41.960397 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:41.960205 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fd7671b332eccf8b2999104569fd648045f321b7e30511b19bcffc3b86ac0ba" Oct 30 20:34:41.960397 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:41.960159 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-fe49e7-cvz2l" Oct 30 20:34:42.581879 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:42.581847 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5379527-3946-433f-8abb-a0d43cc70cde" path="/var/lib/kubelet/pods/a5379527-3946-433f-8abb-a0d43cc70cde/volumes" Oct 30 20:34:43.220920 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:34:43.220885 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:34:50.158727 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:50.158691 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-b37684-zxwlt"] Oct 30 20:34:50.159141 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:50.159121 4292 cpu_manager.go:414] "containerMap: removing container" podUID="db721103-aee1-434a-be3d-ef0d13a54b4d" containerName="revoke-keys" Oct 30 20:34:50.159141 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:50.159132 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db721103-aee1-434a-be3d-ef0d13a54b4d" containerName="revoke-keys" Oct 30 20:34:50.159237 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:50.159196 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="db721103-aee1-434a-be3d-ef0d13a54b4d" containerName="revoke-keys" Oct 30 20:34:50.163665 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:50.163640 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-b37684-zxwlt" Oct 30 20:34:50.216594 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:50.216560 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-b37684-zxwlt"] Oct 30 20:34:50.264596 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:50.264572 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:34:50.264746 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:50.264650 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\"" Oct 30 20:34:50.266079 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:50.266060 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d4b6b54-6d2f-4ffa-9586-e80b29c3295d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-b37684-zxwlt\" (UID: \"5d4b6b54-6d2f-4ffa-9586-e80b29c3295d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-b37684-zxwlt" Oct 30 20:34:50.266129 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:50.266106 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d4b6b54-6d2f-4ffa-9586-e80b29c3295d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-b37684-zxwlt\" (UID: \"5d4b6b54-6d2f-4ffa-9586-e80b29c3295d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-b37684-zxwlt" Oct 30 20:34:50.266198 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:50.266180 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8c5x\" (UniqueName: \"kubernetes.io/projected/5d4b6b54-6d2f-4ffa-9586-e80b29c3295d-kube-api-access-n8c5x\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-b37684-zxwlt\" (UID: \"5d4b6b54-6d2f-4ffa-9586-e80b29c3295d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-b37684-zxwlt" Oct 30 20:34:50.367021 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:50.366989 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d4b6b54-6d2f-4ffa-9586-e80b29c3295d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-b37684-zxwlt\" (UID: \"5d4b6b54-6d2f-4ffa-9586-e80b29c3295d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-b37684-zxwlt" Oct 30 20:34:50.367181 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:50.367043 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d4b6b54-6d2f-4ffa-9586-e80b29c3295d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-b37684-zxwlt\" (UID: \"5d4b6b54-6d2f-4ffa-9586-e80b29c3295d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-b37684-zxwlt" Oct 30 20:34:50.367181 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:50.367091 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8c5x\" (UniqueName: \"kubernetes.io/projected/5d4b6b54-6d2f-4ffa-9586-e80b29c3295d-kube-api-access-n8c5x\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-b37684-zxwlt\" (UID: \"5d4b6b54-6d2f-4ffa-9586-e80b29c3295d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-b37684-zxwlt" Oct 30 20:34:50.367836 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:50.367810 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d4b6b54-6d2f-4ffa-9586-e80b29c3295d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-b37684-zxwlt\" (UID: \"5d4b6b54-6d2f-4ffa-9586-e80b29c3295d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-b37684-zxwlt" Oct 30 20:34:50.369419 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:50.369390 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d4b6b54-6d2f-4ffa-9586-e80b29c3295d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-b37684-zxwlt\" (UID: \"5d4b6b54-6d2f-4ffa-9586-e80b29c3295d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-b37684-zxwlt" Oct 30 20:34:50.430316 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:50.430251 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8c5x\" (UniqueName: \"kubernetes.io/projected/5d4b6b54-6d2f-4ffa-9586-e80b29c3295d-kube-api-access-n8c5x\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-b37684-zxwlt\" (UID: \"5d4b6b54-6d2f-4ffa-9586-e80b29c3295d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-b37684-zxwlt" Oct 30 20:34:50.477218 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:50.477190 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-b37684-zxwlt" Oct 30 20:34:50.857128 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:50.857098 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-b37684-zxwlt"] Oct 30 20:34:50.858112 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:34:50.858087 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d4b6b54_6d2f_4ffa_9586_e80b29c3295d.slice/crio-1741c909116574969f5ff713a6f31c8221bb68dadfd94c8d012017f7123b5f56 WatchSource:0}: Error finding container 1741c909116574969f5ff713a6f31c8221bb68dadfd94c8d012017f7123b5f56: Status 404 returned error can't find the container with id 1741c909116574969f5ff713a6f31c8221bb68dadfd94c8d012017f7123b5f56 Oct 30 20:34:51.013619 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:51.013585 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-b37684-zxwlt" event={"ID":"5d4b6b54-6d2f-4ffa-9586-e80b29c3295d","Type":"ContainerStarted","Data":"1741c909116574969f5ff713a6f31c8221bb68dadfd94c8d012017f7123b5f56"} Oct 30 20:34:52.029577 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:52.029541 4292 generic.go:356] "Generic (PLEG): container finished" podID="5d4b6b54-6d2f-4ffa-9586-e80b29c3295d" containerID="1f0248404cc8b4ad9e06ef6e4ce614dc1d6aa871c6b3a0ceaf2dd957bf984a2f" exitCode=0 Oct 30 20:34:52.030062 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:52.029627 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-b37684-zxwlt" event={"ID":"5d4b6b54-6d2f-4ffa-9586-e80b29c3295d","Type":"ContainerDied","Data":"1f0248404cc8b4ad9e06ef6e4ce614dc1d6aa871c6b3a0ceaf2dd957bf984a2f"} Oct 30 20:34:53.160190 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:53.160167 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-b37684-zxwlt" Oct 30 20:34:53.207558 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:53.207533 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d4b6b54-6d2f-4ffa-9586-e80b29c3295d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d4b6b54-6d2f-4ffa-9586-e80b29c3295d-maas-api-xks-ca\") pod \"5d4b6b54-6d2f-4ffa-9586-e80b29c3295d\" (UID: \"5d4b6b54-6d2f-4ffa-9586-e80b29c3295d\") " Oct 30 20:34:53.207706 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:53.207578 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d4b6b54-6d2f-4ffa-9586-e80b29c3295d-kube-api-access-n8c5x\" (UniqueName: \"kubernetes.io/projected/5d4b6b54-6d2f-4ffa-9586-e80b29c3295d-kube-api-access-n8c5x\") pod \"5d4b6b54-6d2f-4ffa-9586-e80b29c3295d\" (UID: \"5d4b6b54-6d2f-4ffa-9586-e80b29c3295d\") " Oct 30 20:34:53.207776 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:53.207698 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d4b6b54-6d2f-4ffa-9586-e80b29c3295d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d4b6b54-6d2f-4ffa-9586-e80b29c3295d-maas-api-openshift-ca\") pod \"5d4b6b54-6d2f-4ffa-9586-e80b29c3295d\" (UID: \"5d4b6b54-6d2f-4ffa-9586-e80b29c3295d\") " Oct 30 20:34:53.208016 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:53.207989 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d4b6b54-6d2f-4ffa-9586-e80b29c3295d-maas-api-openshift-ca" pod "5d4b6b54-6d2f-4ffa-9586-e80b29c3295d" (UID: "5d4b6b54-6d2f-4ffa-9586-e80b29c3295d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:34:53.209652 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:53.209633 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d4b6b54-6d2f-4ffa-9586-e80b29c3295d-kube-api-access-n8c5x" pod "5d4b6b54-6d2f-4ffa-9586-e80b29c3295d" (UID: "5d4b6b54-6d2f-4ffa-9586-e80b29c3295d"). InnerVolumeSpecName "kube-api-access-n8c5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:34:53.209732 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:53.209632 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d4b6b54-6d2f-4ffa-9586-e80b29c3295d-maas-api-xks-ca" pod "5d4b6b54-6d2f-4ffa-9586-e80b29c3295d" (UID: "5d4b6b54-6d2f-4ffa-9586-e80b29c3295d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:34:53.308662 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:53.308599 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d4b6b54-6d2f-4ffa-9586-e80b29c3295d-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:34:53.308662 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:53.308623 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8c5x\" (UniqueName: \"kubernetes.io/projected/5d4b6b54-6d2f-4ffa-9586-e80b29c3295d-kube-api-access-n8c5x\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:34:53.308662 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:53.308634 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d4b6b54-6d2f-4ffa-9586-e80b29c3295d-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:34:54.043637 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:54.043606 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-b37684-zxwlt" Oct 30 20:34:54.043637 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:54.043616 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-b37684-zxwlt" event={"ID":"5d4b6b54-6d2f-4ffa-9586-e80b29c3295d","Type":"ContainerDied","Data":"1741c909116574969f5ff713a6f31c8221bb68dadfd94c8d012017f7123b5f56"} Oct 30 20:34:54.043821 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:54.043678 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1741c909116574969f5ff713a6f31c8221bb68dadfd94c8d012017f7123b5f56" Oct 30 20:34:58.226135 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:34:58.226106 4292 scope.go:122] "RemoveContainer" containerID="2d88c60b26143ad808580b45258989649da670707c5c28d37964c3a6c2744b8c" Oct 30 20:35:13.220294 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:35:13.220256 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:35:21.890836 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:21.890786 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b91df1-764b746c9f-hz7m2"] Oct 30 20:35:21.891257 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:21.890991 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b91df1-764b746c9f-hz7m2" podUID="527ba566-957f-406d-8756-ce938d0c3922" containerName="maas-api" containerID="cri-o://4d7d9eaef73b4abb2270be44990a74ec18acdee9692f41ea49b3897d603ebc94" gracePeriod=30 Oct 30 20:35:22.197176 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:22.197144 4292 generic.go:356] "Generic (PLEG): container finished" podID="527ba566-957f-406d-8756-ce938d0c3922" containerID="4d7d9eaef73b4abb2270be44990a74ec18acdee9692f41ea49b3897d603ebc94" exitCode=0 Oct 30 20:35:22.197338 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:22.197228 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b91df1-764b746c9f-hz7m2" event={"ID":"527ba566-957f-406d-8756-ce938d0c3922","Type":"ContainerDied","Data":"4d7d9eaef73b4abb2270be44990a74ec18acdee9692f41ea49b3897d603ebc94"} Oct 30 20:35:22.197338 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:22.197297 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b91df1-764b746c9f-hz7m2" event={"ID":"527ba566-957f-406d-8756-ce938d0c3922","Type":"ContainerDied","Data":"8d37d153d369bb51203e2982bfc45c452c7bcee8639b66bed08d93d29aedd2bb"} Oct 30 20:35:22.197338 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:22.197328 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d37d153d369bb51203e2982bfc45c452c7bcee8639b66bed08d93d29aedd2bb" Oct 30 20:35:22.205000 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:22.204974 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b91df1-764b746c9f-hz7m2" Oct 30 20:35:22.309879 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:22.309847 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/527ba566-957f-406d-8756-ce938d0c3922-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/527ba566-957f-406d-8756-ce938d0c3922-maas-api-tls\") pod \"527ba566-957f-406d-8756-ce938d0c3922\" (UID: \"527ba566-957f-406d-8756-ce938d0c3922\") " Oct 30 20:35:22.310114 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:22.309937 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/527ba566-957f-406d-8756-ce938d0c3922-metrics-certs\" (UniqueName: \"kubernetes.io/secret/527ba566-957f-406d-8756-ce938d0c3922-metrics-certs\") pod \"527ba566-957f-406d-8756-ce938d0c3922\" (UID: \"527ba566-957f-406d-8756-ce938d0c3922\") " Oct 30 20:35:22.310114 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:22.309973 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/527ba566-957f-406d-8756-ce938d0c3922-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/527ba566-957f-406d-8756-ce938d0c3922-openshift-service-ca\") pod \"527ba566-957f-406d-8756-ce938d0c3922\" (UID: \"527ba566-957f-406d-8756-ce938d0c3922\") " Oct 30 20:35:22.310114 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:22.310015 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/527ba566-957f-406d-8756-ce938d0c3922-kube-api-access-hv6b4\" (UniqueName: \"kubernetes.io/projected/527ba566-957f-406d-8756-ce938d0c3922-kube-api-access-hv6b4\") pod \"527ba566-957f-406d-8756-ce938d0c3922\" (UID: \"527ba566-957f-406d-8756-ce938d0c3922\") " Oct 30 20:35:22.310407 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:22.310362 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/527ba566-957f-406d-8756-ce938d0c3922-openshift-service-ca" pod "527ba566-957f-406d-8756-ce938d0c3922" (UID: "527ba566-957f-406d-8756-ce938d0c3922"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:35:22.310573 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:22.310520 4292 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/527ba566-957f-406d-8756-ce938d0c3922-openshift-service-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:35:22.312126 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:22.312103 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/527ba566-957f-406d-8756-ce938d0c3922-kube-api-access-hv6b4" pod "527ba566-957f-406d-8756-ce938d0c3922" (UID: "527ba566-957f-406d-8756-ce938d0c3922"). InnerVolumeSpecName "kube-api-access-hv6b4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:35:22.312517 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:22.312500 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/527ba566-957f-406d-8756-ce938d0c3922-metrics-certs" pod "527ba566-957f-406d-8756-ce938d0c3922" (UID: "527ba566-957f-406d-8756-ce938d0c3922"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:35:22.312605 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:22.312571 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/527ba566-957f-406d-8756-ce938d0c3922-maas-api-tls" pod "527ba566-957f-406d-8756-ce938d0c3922" (UID: "527ba566-957f-406d-8756-ce938d0c3922"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:35:22.411450 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:22.411419 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/527ba566-957f-406d-8756-ce938d0c3922-maas-api-tls\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:35:22.411450 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:22.411443 4292 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/527ba566-957f-406d-8756-ce938d0c3922-metrics-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:35:22.411450 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:22.411453 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv6b4\" (UniqueName: \"kubernetes.io/projected/527ba566-957f-406d-8756-ce938d0c3922-kube-api-access-hv6b4\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:35:23.202570 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:23.202528 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b91df1-764b746c9f-hz7m2" Oct 30 20:35:23.372325 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:23.372295 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b91df1-764b746c9f-hz7m2"] Oct 30 20:35:23.422873 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:23.422843 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b91df1-764b746c9f-hz7m2"] Oct 30 20:35:24.581245 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:24.581212 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="527ba566-957f-406d-8756-ce938d0c3922" path="/var/lib/kubelet/pods/527ba566-957f-406d-8756-ce938d0c3922/volumes" Oct 30 20:35:32.815950 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:32.815917 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4ec609-prnk6"] Oct 30 20:35:32.816572 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:32.816553 4292 cpu_manager.go:414] "containerMap: removing container" podUID="5d4b6b54-6d2f-4ffa-9586-e80b29c3295d" containerName="revoke-keys" Oct 30 20:35:32.816672 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:32.816574 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d4b6b54-6d2f-4ffa-9586-e80b29c3295d" containerName="revoke-keys" Oct 30 20:35:32.816672 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:32.816612 4292 cpu_manager.go:414] "containerMap: removing container" podUID="527ba566-957f-406d-8756-ce938d0c3922" containerName="maas-api" Oct 30 20:35:32.816672 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:32.816621 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="527ba566-957f-406d-8756-ce938d0c3922" containerName="maas-api" Oct 30 20:35:32.816830 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:32.816723 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d4b6b54-6d2f-4ffa-9586-e80b29c3295d" containerName="revoke-keys" Oct 30 20:35:32.816830 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:32.816740 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="527ba566-957f-406d-8756-ce938d0c3922" containerName="maas-api" Oct 30 20:35:32.820144 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:32.820124 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4ec609-prnk6" Oct 30 20:35:32.875296 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:32.875264 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4ec609-prnk6"] Oct 30 20:35:32.922333 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:32.922303 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:35:32.922333 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:32.922323 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\"" Oct 30 20:35:32.922562 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:32.922488 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4whb9\" (UniqueName: \"kubernetes.io/projected/6df184b0-58c1-445b-bc48-9146f0075f31-kube-api-access-4whb9\") pod \"maas-api-revoke-keys-e2e-worker-w4-4ec609-prnk6\" (UID: \"6df184b0-58c1-445b-bc48-9146f0075f31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4ec609-prnk6" Oct 30 20:35:32.922643 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:32.922629 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6df184b0-58c1-445b-bc48-9146f0075f31-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-4ec609-prnk6\" (UID: \"6df184b0-58c1-445b-bc48-9146f0075f31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4ec609-prnk6" Oct 30 20:35:32.922718 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:32.922689 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6df184b0-58c1-445b-bc48-9146f0075f31-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-4ec609-prnk6\" (UID: \"6df184b0-58c1-445b-bc48-9146f0075f31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4ec609-prnk6" Oct 30 20:35:33.023797 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:33.023765 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6df184b0-58c1-445b-bc48-9146f0075f31-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-4ec609-prnk6\" (UID: \"6df184b0-58c1-445b-bc48-9146f0075f31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4ec609-prnk6" Oct 30 20:35:33.023964 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:33.023828 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6df184b0-58c1-445b-bc48-9146f0075f31-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-4ec609-prnk6\" (UID: \"6df184b0-58c1-445b-bc48-9146f0075f31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4ec609-prnk6" Oct 30 20:35:33.023964 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:33.023867 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4whb9\" (UniqueName: \"kubernetes.io/projected/6df184b0-58c1-445b-bc48-9146f0075f31-kube-api-access-4whb9\") pod \"maas-api-revoke-keys-e2e-worker-w4-4ec609-prnk6\" (UID: \"6df184b0-58c1-445b-bc48-9146f0075f31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4ec609-prnk6" Oct 30 20:35:33.024530 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:33.024504 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6df184b0-58c1-445b-bc48-9146f0075f31-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-4ec609-prnk6\" (UID: \"6df184b0-58c1-445b-bc48-9146f0075f31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4ec609-prnk6" Oct 30 20:35:33.026152 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:33.026128 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6df184b0-58c1-445b-bc48-9146f0075f31-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-4ec609-prnk6\" (UID: \"6df184b0-58c1-445b-bc48-9146f0075f31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4ec609-prnk6" Oct 30 20:35:33.086128 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:33.086062 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4whb9\" (UniqueName: \"kubernetes.io/projected/6df184b0-58c1-445b-bc48-9146f0075f31-kube-api-access-4whb9\") pod \"maas-api-revoke-keys-e2e-worker-w4-4ec609-prnk6\" (UID: \"6df184b0-58c1-445b-bc48-9146f0075f31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4ec609-prnk6" Oct 30 20:35:33.132876 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:33.132849 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4ec609-prnk6" Oct 30 20:35:33.508388 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:33.508346 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4ec609-prnk6"] Oct 30 20:35:33.510764 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:35:33.510741 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6df184b0_58c1_445b_bc48_9146f0075f31.slice/crio-7ba1c5f6aaed1826fae5857c201cefa9b366fc0b2c95aa621490d48bf245a489 WatchSource:0}: Error finding container 7ba1c5f6aaed1826fae5857c201cefa9b366fc0b2c95aa621490d48bf245a489: Status 404 returned error can't find the container with id 7ba1c5f6aaed1826fae5857c201cefa9b366fc0b2c95aa621490d48bf245a489 Oct 30 20:35:34.262327 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:34.262291 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4ec609-prnk6" event={"ID":"6df184b0-58c1-445b-bc48-9146f0075f31","Type":"ContainerStarted","Data":"a7abcd3032dc4ca6a79406215d50ed52da36c1e59451d1651f2d272e5f12efa5"} Oct 30 20:35:34.262737 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:34.262389 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4ec609-prnk6" event={"ID":"6df184b0-58c1-445b-bc48-9146f0075f31","Type":"ContainerStarted","Data":"7ba1c5f6aaed1826fae5857c201cefa9b366fc0b2c95aa621490d48bf245a489"} Oct 30 20:35:35.271308 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:35.271267 4292 generic.go:356] "Generic (PLEG): container finished" podID="6df184b0-58c1-445b-bc48-9146f0075f31" containerID="a7abcd3032dc4ca6a79406215d50ed52da36c1e59451d1651f2d272e5f12efa5" exitCode=0 Oct 30 20:35:35.271834 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:35.271342 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4ec609-prnk6" event={"ID":"6df184b0-58c1-445b-bc48-9146f0075f31","Type":"ContainerDied","Data":"a7abcd3032dc4ca6a79406215d50ed52da36c1e59451d1651f2d272e5f12efa5"} Oct 30 20:35:35.418107 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:35.418073 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4ec609-prnk6" Oct 30 20:35:35.551157 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:35.551078 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6df184b0-58c1-445b-bc48-9146f0075f31-kube-api-access-4whb9\" (UniqueName: \"kubernetes.io/projected/6df184b0-58c1-445b-bc48-9146f0075f31-kube-api-access-4whb9\") pod \"6df184b0-58c1-445b-bc48-9146f0075f31\" (UID: \"6df184b0-58c1-445b-bc48-9146f0075f31\") " Oct 30 20:35:35.551157 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:35.551115 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6df184b0-58c1-445b-bc48-9146f0075f31-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6df184b0-58c1-445b-bc48-9146f0075f31-maas-api-openshift-ca\") pod \"6df184b0-58c1-445b-bc48-9146f0075f31\" (UID: \"6df184b0-58c1-445b-bc48-9146f0075f31\") " Oct 30 20:35:35.551157 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:35.551150 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6df184b0-58c1-445b-bc48-9146f0075f31-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6df184b0-58c1-445b-bc48-9146f0075f31-maas-api-xks-ca\") pod \"6df184b0-58c1-445b-bc48-9146f0075f31\" (UID: \"6df184b0-58c1-445b-bc48-9146f0075f31\") " Oct 30 20:35:35.551544 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:35.551516 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6df184b0-58c1-445b-bc48-9146f0075f31-maas-api-openshift-ca" pod "6df184b0-58c1-445b-bc48-9146f0075f31" (UID: "6df184b0-58c1-445b-bc48-9146f0075f31"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:35:35.551688 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:35.551657 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6df184b0-58c1-445b-bc48-9146f0075f31-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:35:35.553573 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:35.553545 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6df184b0-58c1-445b-bc48-9146f0075f31-kube-api-access-4whb9" pod "6df184b0-58c1-445b-bc48-9146f0075f31" (UID: "6df184b0-58c1-445b-bc48-9146f0075f31"). InnerVolumeSpecName "kube-api-access-4whb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:35:35.559489 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:35.559459 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6df184b0-58c1-445b-bc48-9146f0075f31-maas-api-xks-ca" pod "6df184b0-58c1-445b-bc48-9146f0075f31" (UID: "6df184b0-58c1-445b-bc48-9146f0075f31"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:35:35.652407 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:35.652378 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4whb9\" (UniqueName: \"kubernetes.io/projected/6df184b0-58c1-445b-bc48-9146f0075f31-kube-api-access-4whb9\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:35:35.652407 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:35.652404 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6df184b0-58c1-445b-bc48-9146f0075f31-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:35:36.278476 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:36.278445 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4ec609-prnk6" Oct 30 20:35:36.278476 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:36.278471 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4ec609-prnk6" event={"ID":"6df184b0-58c1-445b-bc48-9146f0075f31","Type":"ContainerDied","Data":"7ba1c5f6aaed1826fae5857c201cefa9b366fc0b2c95aa621490d48bf245a489"} Oct 30 20:35:36.279047 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:36.278514 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ba1c5f6aaed1826fae5857c201cefa9b366fc0b2c95aa621490d48bf245a489" Oct 30 20:35:37.236138 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:37.236104 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4ec609/premium-w4-4ec609-kserve-69f7c88bf7-rzmmt"] Oct 30 20:35:41.953796 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:41.953763 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0e72bd-h8vxx"] Oct 30 20:35:42.007609 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:42.007572 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0e72bd-h8vxx"] Oct 30 20:35:42.581813 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:42.581779 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f3a292e-96b6-411b-9503-1b5a74c7f5ab" path="/var/lib/kubelet/pods/4f3a292e-96b6-411b-9503-1b5a74c7f5ab/volumes" Oct 30 20:35:43.220764 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:35:43.220731 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:35:49.097708 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:49.097674 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds"] Oct 30 20:35:49.098243 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:49.097880 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds" podUID="03c3a9f1-5156-41a2-a643-46343f91b78e" containerName="istio-proxy" containerID="cri-o://ca5f1572b9c2328c499c179600bf48fd0b6f62c20d6f6ea92cac3847e446678d" gracePeriod=30 Oct 30 20:35:49.924302 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:49.924268 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e391de-z9ghb"] Oct 30 20:35:49.924747 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:49.924733 4292 cpu_manager.go:414] "containerMap: removing container" podUID="6df184b0-58c1-445b-bc48-9146f0075f31" containerName="revoke-keys" Oct 30 20:35:49.924792 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:49.924749 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6df184b0-58c1-445b-bc48-9146f0075f31" containerName="revoke-keys" Oct 30 20:35:49.924828 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:49.924817 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="6df184b0-58c1-445b-bc48-9146f0075f31" containerName="revoke-keys" Oct 30 20:35:49.928246 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:49.928231 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e391de-z9ghb" Oct 30 20:35:49.981988 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:49.981960 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:35:49.983474 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:49.983447 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e391de-z9ghb"] Oct 30 20:35:50.015998 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:50.015966 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/720c0839-9035-4182-a45c-0fc081686e0c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-e391de-z9ghb\" (UID: \"720c0839-9035-4182-a45c-0fc081686e0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e391de-z9ghb" Oct 30 20:35:50.016163 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:50.016096 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/720c0839-9035-4182-a45c-0fc081686e0c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-e391de-z9ghb\" (UID: \"720c0839-9035-4182-a45c-0fc081686e0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e391de-z9ghb" Oct 30 20:35:50.016233 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:50.016192 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9xk5\" (UniqueName: \"kubernetes.io/projected/720c0839-9035-4182-a45c-0fc081686e0c-kube-api-access-k9xk5\") pod \"maas-api-revoke-keys-e2e-extmod-b-e391de-z9ghb\" (UID: \"720c0839-9035-4182-a45c-0fc081686e0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e391de-z9ghb" Oct 30 20:35:50.031228 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:50.031206 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\"" Oct 30 20:35:50.117116 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:50.117088 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/720c0839-9035-4182-a45c-0fc081686e0c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-e391de-z9ghb\" (UID: \"720c0839-9035-4182-a45c-0fc081686e0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e391de-z9ghb" Oct 30 20:35:50.117512 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:50.117188 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/720c0839-9035-4182-a45c-0fc081686e0c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-e391de-z9ghb\" (UID: \"720c0839-9035-4182-a45c-0fc081686e0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e391de-z9ghb" Oct 30 20:35:50.117512 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:50.117241 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9xk5\" (UniqueName: \"kubernetes.io/projected/720c0839-9035-4182-a45c-0fc081686e0c-kube-api-access-k9xk5\") pod \"maas-api-revoke-keys-e2e-extmod-b-e391de-z9ghb\" (UID: \"720c0839-9035-4182-a45c-0fc081686e0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e391de-z9ghb" Oct 30 20:35:50.117817 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:50.117797 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/720c0839-9035-4182-a45c-0fc081686e0c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-e391de-z9ghb\" (UID: \"720c0839-9035-4182-a45c-0fc081686e0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e391de-z9ghb" Oct 30 20:35:50.119484 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:50.119468 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/720c0839-9035-4182-a45c-0fc081686e0c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-e391de-z9ghb\" (UID: \"720c0839-9035-4182-a45c-0fc081686e0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e391de-z9ghb" Oct 30 20:35:50.180537 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:50.180452 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9xk5\" (UniqueName: \"kubernetes.io/projected/720c0839-9035-4182-a45c-0fc081686e0c-kube-api-access-k9xk5\") pod \"maas-api-revoke-keys-e2e-extmod-b-e391de-z9ghb\" (UID: \"720c0839-9035-4182-a45c-0fc081686e0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e391de-z9ghb" Oct 30 20:35:50.240588 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:50.240552 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e391de-z9ghb" Oct 30 20:35:50.613785 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:35:50.613753 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod720c0839_9035_4182_a45c_0fc081686e0c.slice/crio-acf1fc4ba983c029989f6fc057f9b5e81b312988e4d7074fe843b7ab5418eaac WatchSource:0}: Error finding container acf1fc4ba983c029989f6fc057f9b5e81b312988e4d7074fe843b7ab5418eaac: Status 404 returned error can't find the container with id acf1fc4ba983c029989f6fc057f9b5e81b312988e4d7074fe843b7ab5418eaac Oct 30 20:35:50.616700 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:50.616677 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e391de-z9ghb"] Oct 30 20:35:51.377294 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:51.377258 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e391de-z9ghb" event={"ID":"720c0839-9035-4182-a45c-0fc081686e0c","Type":"ContainerStarted","Data":"8f57066d64f8a5caa0c562589922a9a98ea16a1638188e7fa831005c85d8a164"} Oct 30 20:35:51.377788 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:51.377319 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e391de-z9ghb" event={"ID":"720c0839-9035-4182-a45c-0fc081686e0c","Type":"ContainerStarted","Data":"acf1fc4ba983c029989f6fc057f9b5e81b312988e4d7074fe843b7ab5418eaac"} Oct 30 20:35:51.547122 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:51.547073 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e391de-z9ghb" podStartSLOduration=2.547058322 podStartE2EDuration="2.547058322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:35:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:35:51.49585048 +0000 UTC m=+4255.014165935" watchObservedRunningTime="2026-10-30 20:35:51.547058322 +0000 UTC m=+4255.065373779" Oct 30 20:35:52.255579 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:52.255538 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-4ec609/premium-w4-4ec609-kserve-69f7c88bf7-rzmmt" podUID="fb6be431-0a6c-4d81-b365-d7537f5d385a" containerName="main" containerID="cri-o://85a25601e116c5f2a5baa471fe6a6af436c3b2c8ea9136b98009fe3e9db74cf6" gracePeriod=45 Oct 30 20:35:52.384635 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:52.384600 4292 generic.go:356] "Generic (PLEG): container finished" podID="fb6be431-0a6c-4d81-b365-d7537f5d385a" containerID="85a25601e116c5f2a5baa471fe6a6af436c3b2c8ea9136b98009fe3e9db74cf6" exitCode=0 Oct 30 20:35:52.385043 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:52.384667 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ec609/premium-w4-4ec609-kserve-69f7c88bf7-rzmmt" event={"ID":"fb6be431-0a6c-4d81-b365-d7537f5d385a","Type":"ContainerDied","Data":"85a25601e116c5f2a5baa471fe6a6af436c3b2c8ea9136b98009fe3e9db74cf6"} Oct 30 20:35:52.386565 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:52.386527 4292 generic.go:356] "Generic (PLEG): container finished" podID="720c0839-9035-4182-a45c-0fc081686e0c" containerID="8f57066d64f8a5caa0c562589922a9a98ea16a1638188e7fa831005c85d8a164" exitCode=0 Oct 30 20:35:52.386695 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:52.386608 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e391de-z9ghb" event={"ID":"720c0839-9035-4182-a45c-0fc081686e0c","Type":"ContainerDied","Data":"8f57066d64f8a5caa0c562589922a9a98ea16a1638188e7fa831005c85d8a164"} Oct 30 20:35:52.506637 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:52.506583 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ec609/premium-w4-4ec609-kserve-69f7c88bf7-rzmmt" Oct 30 20:35:52.643523 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:52.643489 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb6be431-0a6c-4d81-b365-d7537f5d385a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb6be431-0a6c-4d81-b365-d7537f5d385a-dshm\") pod \"fb6be431-0a6c-4d81-b365-d7537f5d385a\" (UID: \"fb6be431-0a6c-4d81-b365-d7537f5d385a\") " Oct 30 20:35:52.643711 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:52.643562 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb6be431-0a6c-4d81-b365-d7537f5d385a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb6be431-0a6c-4d81-b365-d7537f5d385a-tmp-dir\") pod \"fb6be431-0a6c-4d81-b365-d7537f5d385a\" (UID: \"fb6be431-0a6c-4d81-b365-d7537f5d385a\") " Oct 30 20:35:52.643711 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:52.643625 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb6be431-0a6c-4d81-b365-d7537f5d385a-tls-certs\" (UniqueName: \"kubernetes.io/secret/fb6be431-0a6c-4d81-b365-d7537f5d385a-tls-certs\") pod \"fb6be431-0a6c-4d81-b365-d7537f5d385a\" (UID: \"fb6be431-0a6c-4d81-b365-d7537f5d385a\") " Oct 30 20:35:52.643711 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:52.643669 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb6be431-0a6c-4d81-b365-d7537f5d385a-home\" (UniqueName: \"kubernetes.io/empty-dir/fb6be431-0a6c-4d81-b365-d7537f5d385a-home\") pod \"fb6be431-0a6c-4d81-b365-d7537f5d385a\" (UID: \"fb6be431-0a6c-4d81-b365-d7537f5d385a\") " Oct 30 20:35:52.643877 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:52.643753 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb6be431-0a6c-4d81-b365-d7537f5d385a-kube-api-access-lhtnz\" (UniqueName: \"kubernetes.io/projected/fb6be431-0a6c-4d81-b365-d7537f5d385a-kube-api-access-lhtnz\") pod \"fb6be431-0a6c-4d81-b365-d7537f5d385a\" (UID: \"fb6be431-0a6c-4d81-b365-d7537f5d385a\") " Oct 30 20:35:52.643877 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:52.643807 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb6be431-0a6c-4d81-b365-d7537f5d385a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb6be431-0a6c-4d81-b365-d7537f5d385a-model-cache\") pod \"fb6be431-0a6c-4d81-b365-d7537f5d385a\" (UID: \"fb6be431-0a6c-4d81-b365-d7537f5d385a\") " Oct 30 20:35:52.644832 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:52.644806 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb6be431-0a6c-4d81-b365-d7537f5d385a-home" pod "fb6be431-0a6c-4d81-b365-d7537f5d385a" (UID: "fb6be431-0a6c-4d81-b365-d7537f5d385a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:35:52.645600 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:52.645572 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb6be431-0a6c-4d81-b365-d7537f5d385a-model-cache" pod "fb6be431-0a6c-4d81-b365-d7537f5d385a" (UID: "fb6be431-0a6c-4d81-b365-d7537f5d385a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:35:52.645722 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:52.645700 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb6be431-0a6c-4d81-b365-d7537f5d385a-tmp-dir" pod "fb6be431-0a6c-4d81-b365-d7537f5d385a" (UID: "fb6be431-0a6c-4d81-b365-d7537f5d385a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:35:52.647542 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:52.647512 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb6be431-0a6c-4d81-b365-d7537f5d385a-tls-certs" pod "fb6be431-0a6c-4d81-b365-d7537f5d385a" (UID: "fb6be431-0a6c-4d81-b365-d7537f5d385a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:35:52.647542 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:52.647525 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb6be431-0a6c-4d81-b365-d7537f5d385a-dshm" pod "fb6be431-0a6c-4d81-b365-d7537f5d385a" (UID: "fb6be431-0a6c-4d81-b365-d7537f5d385a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:35:52.648027 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:52.648001 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb6be431-0a6c-4d81-b365-d7537f5d385a-kube-api-access-lhtnz" pod "fb6be431-0a6c-4d81-b365-d7537f5d385a" (UID: "fb6be431-0a6c-4d81-b365-d7537f5d385a"). InnerVolumeSpecName "kube-api-access-lhtnz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:35:52.745727 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:52.745694 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhtnz\" (UniqueName: \"kubernetes.io/projected/fb6be431-0a6c-4d81-b365-d7537f5d385a-kube-api-access-lhtnz\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:35:52.745727 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:52.745725 4292 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb6be431-0a6c-4d81-b365-d7537f5d385a-model-cache\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:35:52.745727 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:52.745735 4292 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb6be431-0a6c-4d81-b365-d7537f5d385a-dshm\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:35:52.745965 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:52.745765 4292 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb6be431-0a6c-4d81-b365-d7537f5d385a-tmp-dir\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:35:52.745965 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:52.745774 4292 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb6be431-0a6c-4d81-b365-d7537f5d385a-tls-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:35:52.745965 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:52.745782 4292 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb6be431-0a6c-4d81-b365-d7537f5d385a-home\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:35:53.393450 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:53.393417 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ec609/premium-w4-4ec609-kserve-69f7c88bf7-rzmmt" Oct 30 20:35:53.393919 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:53.393450 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ec609/premium-w4-4ec609-kserve-69f7c88bf7-rzmmt" event={"ID":"fb6be431-0a6c-4d81-b365-d7537f5d385a","Type":"ContainerDied","Data":"81c9b83d4a5be10cf3c74d94d1f56b275063c5611034afde5af78ca472d7704d"} Oct 30 20:35:53.393919 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:53.393501 4292 scope.go:122] "RemoveContainer" containerID="85a25601e116c5f2a5baa471fe6a6af436c3b2c8ea9136b98009fe3e9db74cf6" Oct 30 20:35:53.522079 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:53.522056 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e391de-z9ghb" Oct 30 20:35:53.562892 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:53.562861 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4ec609/premium-w4-4ec609-kserve-69f7c88bf7-rzmmt"] Oct 30 20:35:53.613313 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:53.613282 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-4ec609/premium-w4-4ec609-kserve-69f7c88bf7-rzmmt"] Oct 30 20:35:53.656384 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:53.656284 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/720c0839-9035-4182-a45c-0fc081686e0c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/720c0839-9035-4182-a45c-0fc081686e0c-maas-api-xks-ca\") pod \"720c0839-9035-4182-a45c-0fc081686e0c\" (UID: \"720c0839-9035-4182-a45c-0fc081686e0c\") " Oct 30 20:35:53.656384 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:53.656323 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/720c0839-9035-4182-a45c-0fc081686e0c-kube-api-access-k9xk5\" (UniqueName: \"kubernetes.io/projected/720c0839-9035-4182-a45c-0fc081686e0c-kube-api-access-k9xk5\") pod \"720c0839-9035-4182-a45c-0fc081686e0c\" (UID: \"720c0839-9035-4182-a45c-0fc081686e0c\") " Oct 30 20:35:53.656611 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:53.656416 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/720c0839-9035-4182-a45c-0fc081686e0c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/720c0839-9035-4182-a45c-0fc081686e0c-maas-api-openshift-ca\") pod \"720c0839-9035-4182-a45c-0fc081686e0c\" (UID: \"720c0839-9035-4182-a45c-0fc081686e0c\") " Oct 30 20:35:53.656828 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:53.656797 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/720c0839-9035-4182-a45c-0fc081686e0c-maas-api-openshift-ca" pod "720c0839-9035-4182-a45c-0fc081686e0c" (UID: "720c0839-9035-4182-a45c-0fc081686e0c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:35:53.658682 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:53.658661 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/720c0839-9035-4182-a45c-0fc081686e0c-kube-api-access-k9xk5" pod "720c0839-9035-4182-a45c-0fc081686e0c" (UID: "720c0839-9035-4182-a45c-0fc081686e0c"). InnerVolumeSpecName "kube-api-access-k9xk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:35:53.658801 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:53.658789 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/720c0839-9035-4182-a45c-0fc081686e0c-maas-api-xks-ca" pod "720c0839-9035-4182-a45c-0fc081686e0c" (UID: "720c0839-9035-4182-a45c-0fc081686e0c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:35:53.757752 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:53.757715 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/720c0839-9035-4182-a45c-0fc081686e0c-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:35:53.757752 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:53.757746 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9xk5\" (UniqueName: \"kubernetes.io/projected/720c0839-9035-4182-a45c-0fc081686e0c-kube-api-access-k9xk5\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:35:53.757752 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:53.757759 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/720c0839-9035-4182-a45c-0fc081686e0c-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:35:54.343113 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:54.343088 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds" Oct 30 20:35:54.400706 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:54.400622 4292 generic.go:356] "Generic (PLEG): container finished" podID="03c3a9f1-5156-41a2-a643-46343f91b78e" containerID="ca5f1572b9c2328c499c179600bf48fd0b6f62c20d6f6ea92cac3847e446678d" exitCode=0 Oct 30 20:35:54.400706 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:54.400696 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds" Oct 30 20:35:54.401198 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:54.400744 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds" event={"ID":"03c3a9f1-5156-41a2-a643-46343f91b78e","Type":"ContainerDied","Data":"ca5f1572b9c2328c499c179600bf48fd0b6f62c20d6f6ea92cac3847e446678d"} Oct 30 20:35:54.401198 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:54.400807 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds" event={"ID":"03c3a9f1-5156-41a2-a643-46343f91b78e","Type":"ContainerDied","Data":"718bd3be22fcd0be3bcf822a7fd28fe3b98a38615c16da8b75d3db8b72bf373b"} Oct 30 20:35:54.401198 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:54.400842 4292 scope.go:122] "RemoveContainer" containerID="ca5f1572b9c2328c499c179600bf48fd0b6f62c20d6f6ea92cac3847e446678d" Oct 30 20:35:54.404027 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:54.404005 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e391de-z9ghb" event={"ID":"720c0839-9035-4182-a45c-0fc081686e0c","Type":"ContainerDied","Data":"acf1fc4ba983c029989f6fc057f9b5e81b312988e4d7074fe843b7ab5418eaac"} Oct 30 20:35:54.404175 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:54.404036 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e391de-z9ghb" Oct 30 20:35:54.404175 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:54.404045 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acf1fc4ba983c029989f6fc057f9b5e81b312988e4d7074fe843b7ab5418eaac" Oct 30 20:35:54.411517 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:54.411497 4292 scope.go:122] "RemoveContainer" containerID="f61b08f353f0f4e288f4c040cd0e9ab5164fe2efc409f8a2bf1c0d2aaecfac32" Oct 30 20:35:54.422931 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:54.422915 4292 scope.go:122] "RemoveContainer" containerID="ca5f1572b9c2328c499c179600bf48fd0b6f62c20d6f6ea92cac3847e446678d" Oct 30 20:35:54.423187 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:35:54.423168 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca5f1572b9c2328c499c179600bf48fd0b6f62c20d6f6ea92cac3847e446678d\": container with ID starting with ca5f1572b9c2328c499c179600bf48fd0b6f62c20d6f6ea92cac3847e446678d not found: ID does not exist" containerID="ca5f1572b9c2328c499c179600bf48fd0b6f62c20d6f6ea92cac3847e446678d" Oct 30 20:35:54.423275 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:54.423197 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca5f1572b9c2328c499c179600bf48fd0b6f62c20d6f6ea92cac3847e446678d"} err="failed to get container status \"ca5f1572b9c2328c499c179600bf48fd0b6f62c20d6f6ea92cac3847e446678d\": rpc error: code = NotFound desc = could not find container \"ca5f1572b9c2328c499c179600bf48fd0b6f62c20d6f6ea92cac3847e446678d\": container with ID starting with ca5f1572b9c2328c499c179600bf48fd0b6f62c20d6f6ea92cac3847e446678d not found: ID does not exist" Oct 30 20:35:54.423275 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:54.423223 4292 scope.go:122] "RemoveContainer" containerID="f61b08f353f0f4e288f4c040cd0e9ab5164fe2efc409f8a2bf1c0d2aaecfac32" Oct 30 20:35:54.423510 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:35:54.423493 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f61b08f353f0f4e288f4c040cd0e9ab5164fe2efc409f8a2bf1c0d2aaecfac32\": container with ID starting with f61b08f353f0f4e288f4c040cd0e9ab5164fe2efc409f8a2bf1c0d2aaecfac32 not found: ID does not exist" containerID="f61b08f353f0f4e288f4c040cd0e9ab5164fe2efc409f8a2bf1c0d2aaecfac32" Oct 30 20:35:54.423554 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:54.423517 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f61b08f353f0f4e288f4c040cd0e9ab5164fe2efc409f8a2bf1c0d2aaecfac32"} err="failed to get container status \"f61b08f353f0f4e288f4c040cd0e9ab5164fe2efc409f8a2bf1c0d2aaecfac32\": rpc error: code = NotFound desc = could not find container \"f61b08f353f0f4e288f4c040cd0e9ab5164fe2efc409f8a2bf1c0d2aaecfac32\": container with ID starting with f61b08f353f0f4e288f4c040cd0e9ab5164fe2efc409f8a2bf1c0d2aaecfac32 not found: ID does not exist" Oct 30 20:35:54.465001 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:54.464975 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03c3a9f1-5156-41a2-a643-46343f91b78e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03c3a9f1-5156-41a2-a643-46343f91b78e-credential-socket\") pod \"03c3a9f1-5156-41a2-a643-46343f91b78e\" (UID: \"03c3a9f1-5156-41a2-a643-46343f91b78e\") " Oct 30 20:35:54.465117 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:54.465031 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03c3a9f1-5156-41a2-a643-46343f91b78e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03c3a9f1-5156-41a2-a643-46343f91b78e-istio-envoy\") pod \"03c3a9f1-5156-41a2-a643-46343f91b78e\" (UID: \"03c3a9f1-5156-41a2-a643-46343f91b78e\") " Oct 30 20:35:54.465117 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:54.465065 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03c3a9f1-5156-41a2-a643-46343f91b78e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03c3a9f1-5156-41a2-a643-46343f91b78e-workload-socket\") pod \"03c3a9f1-5156-41a2-a643-46343f91b78e\" (UID: \"03c3a9f1-5156-41a2-a643-46343f91b78e\") " Oct 30 20:35:54.465117 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:54.465092 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03c3a9f1-5156-41a2-a643-46343f91b78e-kube-api-access-j2pd4\" (UniqueName: \"kubernetes.io/projected/03c3a9f1-5156-41a2-a643-46343f91b78e-kube-api-access-j2pd4\") pod \"03c3a9f1-5156-41a2-a643-46343f91b78e\" (UID: \"03c3a9f1-5156-41a2-a643-46343f91b78e\") " Oct 30 20:35:54.465117 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:54.465111 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03c3a9f1-5156-41a2-a643-46343f91b78e-istio-token\" (UniqueName: \"kubernetes.io/projected/03c3a9f1-5156-41a2-a643-46343f91b78e-istio-token\") pod \"03c3a9f1-5156-41a2-a643-46343f91b78e\" (UID: \"03c3a9f1-5156-41a2-a643-46343f91b78e\") " Oct 30 20:35:54.465335 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:54.465139 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03c3a9f1-5156-41a2-a643-46343f91b78e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03c3a9f1-5156-41a2-a643-46343f91b78e-istiod-ca-cert\") pod \"03c3a9f1-5156-41a2-a643-46343f91b78e\" (UID: \"03c3a9f1-5156-41a2-a643-46343f91b78e\") " Oct 30 20:35:54.465335 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:54.465229 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03c3a9f1-5156-41a2-a643-46343f91b78e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03c3a9f1-5156-41a2-a643-46343f91b78e-istio-data\") pod \"03c3a9f1-5156-41a2-a643-46343f91b78e\" (UID: \"03c3a9f1-5156-41a2-a643-46343f91b78e\") " Oct 30 20:35:54.465335 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:54.465250 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03c3a9f1-5156-41a2-a643-46343f91b78e-credential-socket" pod "03c3a9f1-5156-41a2-a643-46343f91b78e" (UID: "03c3a9f1-5156-41a2-a643-46343f91b78e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:35:54.465335 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:54.465267 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03c3a9f1-5156-41a2-a643-46343f91b78e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03c3a9f1-5156-41a2-a643-46343f91b78e-workload-certs\") pod \"03c3a9f1-5156-41a2-a643-46343f91b78e\" (UID: \"03c3a9f1-5156-41a2-a643-46343f91b78e\") " Oct 30 20:35:54.465335 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:54.465314 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03c3a9f1-5156-41a2-a643-46343f91b78e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03c3a9f1-5156-41a2-a643-46343f91b78e-istio-podinfo\") pod \"03c3a9f1-5156-41a2-a643-46343f91b78e\" (UID: \"03c3a9f1-5156-41a2-a643-46343f91b78e\") " Oct 30 20:35:54.465648 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:54.465393 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03c3a9f1-5156-41a2-a643-46343f91b78e-workload-socket" pod "03c3a9f1-5156-41a2-a643-46343f91b78e" (UID: "03c3a9f1-5156-41a2-a643-46343f91b78e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:35:54.465707 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:54.465652 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03c3a9f1-5156-41a2-a643-46343f91b78e-workload-certs" pod "03c3a9f1-5156-41a2-a643-46343f91b78e" (UID: "03c3a9f1-5156-41a2-a643-46343f91b78e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:35:54.465707 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:54.465670 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03c3a9f1-5156-41a2-a643-46343f91b78e-istiod-ca-cert" pod "03c3a9f1-5156-41a2-a643-46343f91b78e" (UID: "03c3a9f1-5156-41a2-a643-46343f91b78e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:35:54.465835 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:54.465815 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03c3a9f1-5156-41a2-a643-46343f91b78e-credential-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:35:54.465898 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:54.465842 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03c3a9f1-5156-41a2-a643-46343f91b78e-workload-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:35:54.465898 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:54.465858 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03c3a9f1-5156-41a2-a643-46343f91b78e-istiod-ca-cert\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:35:54.465898 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:54.465857 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03c3a9f1-5156-41a2-a643-46343f91b78e-istio-data" pod "03c3a9f1-5156-41a2-a643-46343f91b78e" (UID: "03c3a9f1-5156-41a2-a643-46343f91b78e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:35:54.466033 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:54.465876 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03c3a9f1-5156-41a2-a643-46343f91b78e-workload-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:35:54.467722 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:54.467704 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03c3a9f1-5156-41a2-a643-46343f91b78e-istio-envoy" pod "03c3a9f1-5156-41a2-a643-46343f91b78e" (UID: "03c3a9f1-5156-41a2-a643-46343f91b78e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:35:54.467794 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:54.467736 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03c3a9f1-5156-41a2-a643-46343f91b78e-kube-api-access-j2pd4" pod "03c3a9f1-5156-41a2-a643-46343f91b78e" (UID: "03c3a9f1-5156-41a2-a643-46343f91b78e"). InnerVolumeSpecName "kube-api-access-j2pd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:35:54.467886 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:54.467865 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03c3a9f1-5156-41a2-a643-46343f91b78e-istio-podinfo" pod "03c3a9f1-5156-41a2-a643-46343f91b78e" (UID: "03c3a9f1-5156-41a2-a643-46343f91b78e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:35:54.467928 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:54.467877 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03c3a9f1-5156-41a2-a643-46343f91b78e-istio-token" pod "03c3a9f1-5156-41a2-a643-46343f91b78e" (UID: "03c3a9f1-5156-41a2-a643-46343f91b78e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:35:54.566746 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:54.566715 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2pd4\" (UniqueName: \"kubernetes.io/projected/03c3a9f1-5156-41a2-a643-46343f91b78e-kube-api-access-j2pd4\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:35:54.566746 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:54.566746 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03c3a9f1-5156-41a2-a643-46343f91b78e-istio-token\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:35:54.566946 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:54.566760 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03c3a9f1-5156-41a2-a643-46343f91b78e-istio-data\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:35:54.566946 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:54.566772 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03c3a9f1-5156-41a2-a643-46343f91b78e-istio-podinfo\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:35:54.566946 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:54.566785 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03c3a9f1-5156-41a2-a643-46343f91b78e-istio-envoy\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:35:54.582097 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:54.582069 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb6be431-0a6c-4d81-b365-d7537f5d385a" path="/var/lib/kubelet/pods/fb6be431-0a6c-4d81-b365-d7537f5d385a/volumes" Oct 30 20:35:54.818921 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:54.818888 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds"] Oct 30 20:35:54.825015 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:54.824985 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-0e72bd-openshift-default-ff466bf5c-xmjds"] Oct 30 20:35:56.582298 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:56.582262 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03c3a9f1-5156-41a2-a643-46343f91b78e" path="/var/lib/kubelet/pods/03c3a9f1-5156-41a2-a643-46343f91b78e/volumes" Oct 30 20:35:58.263374 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:58.263323 4292 scope.go:122] "RemoveContainer" containerID="53420a01c5c04acfe36b84d7e2079e3d7cbe40037517b42b922582e6e2982b7f" Oct 30 20:35:59.602850 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:59.602814 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-fe49e7-cvz2l"] Oct 30 20:35:59.653466 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:35:59.653439 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-fe49e7-cvz2l"] Oct 30 20:36:00.581727 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:00.581696 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db721103-aee1-434a-be3d-ef0d13a54b4d" path="/var/lib/kubelet/pods/db721103-aee1-434a-be3d-ef0d13a54b4d/volumes" Oct 30 20:36:07.598576 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:07.598530 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bbfa07-qgnvt"] Oct 30 20:36:07.598997 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:07.598985 4292 cpu_manager.go:414] "containerMap: removing container" podUID="03c3a9f1-5156-41a2-a643-46343f91b78e" containerName="istio-proxy" Oct 30 20:36:07.599060 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:07.598998 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03c3a9f1-5156-41a2-a643-46343f91b78e" containerName="istio-proxy" Oct 30 20:36:07.599060 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:07.599012 4292 cpu_manager.go:414] "containerMap: removing container" podUID="03c3a9f1-5156-41a2-a643-46343f91b78e" containerName="istio-proxy" Oct 30 20:36:07.599060 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:07.599017 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03c3a9f1-5156-41a2-a643-46343f91b78e" containerName="istio-proxy" Oct 30 20:36:07.599060 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:07.599030 4292 cpu_manager.go:414] "containerMap: removing container" podUID="720c0839-9035-4182-a45c-0fc081686e0c" containerName="revoke-keys" Oct 30 20:36:07.599060 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:07.599035 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="720c0839-9035-4182-a45c-0fc081686e0c" containerName="revoke-keys" Oct 30 20:36:07.599060 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:07.599045 4292 cpu_manager.go:414] "containerMap: removing container" podUID="fb6be431-0a6c-4d81-b365-d7537f5d385a" containerName="main" Oct 30 20:36:07.599060 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:07.599050 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb6be431-0a6c-4d81-b365-d7537f5d385a" containerName="main" Oct 30 20:36:07.599298 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:07.599104 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb6be431-0a6c-4d81-b365-d7537f5d385a" containerName="main" Oct 30 20:36:07.599298 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:07.599117 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="03c3a9f1-5156-41a2-a643-46343f91b78e" containerName="istio-proxy" Oct 30 20:36:07.599298 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:07.599122 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="03c3a9f1-5156-41a2-a643-46343f91b78e" containerName="istio-proxy" Oct 30 20:36:07.599298 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:07.599128 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="720c0839-9035-4182-a45c-0fc081686e0c" containerName="revoke-keys" Oct 30 20:36:07.602088 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:07.602072 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bbfa07-qgnvt" Oct 30 20:36:07.661274 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:07.661219 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bbfa07-qgnvt"] Oct 30 20:36:07.702921 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:07.702890 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0de01d4b-467d-4d84-8f5d-aca0db0fba42-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-bbfa07-qgnvt\" (UID: \"0de01d4b-467d-4d84-8f5d-aca0db0fba42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bbfa07-qgnvt" Oct 30 20:36:07.703055 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:07.703036 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0de01d4b-467d-4d84-8f5d-aca0db0fba42-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-bbfa07-qgnvt\" (UID: \"0de01d4b-467d-4d84-8f5d-aca0db0fba42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bbfa07-qgnvt" Oct 30 20:36:07.703109 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:07.703071 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxk2f\" (UniqueName: \"kubernetes.io/projected/0de01d4b-467d-4d84-8f5d-aca0db0fba42-kube-api-access-wxk2f\") pod \"maas-api-revoke-keys-e2e-worker-w3-bbfa07-qgnvt\" (UID: \"0de01d4b-467d-4d84-8f5d-aca0db0fba42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bbfa07-qgnvt" Oct 30 20:36:07.707648 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:07.707628 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:36:07.707760 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:07.707703 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\"" Oct 30 20:36:07.803828 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:07.803796 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0de01d4b-467d-4d84-8f5d-aca0db0fba42-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-bbfa07-qgnvt\" (UID: \"0de01d4b-467d-4d84-8f5d-aca0db0fba42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bbfa07-qgnvt" Oct 30 20:36:07.803828 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:07.803831 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxk2f\" (UniqueName: \"kubernetes.io/projected/0de01d4b-467d-4d84-8f5d-aca0db0fba42-kube-api-access-wxk2f\") pod \"maas-api-revoke-keys-e2e-worker-w3-bbfa07-qgnvt\" (UID: \"0de01d4b-467d-4d84-8f5d-aca0db0fba42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bbfa07-qgnvt" Oct 30 20:36:07.804068 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:07.803867 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0de01d4b-467d-4d84-8f5d-aca0db0fba42-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-bbfa07-qgnvt\" (UID: \"0de01d4b-467d-4d84-8f5d-aca0db0fba42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bbfa07-qgnvt" Oct 30 20:36:07.804476 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:07.804454 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0de01d4b-467d-4d84-8f5d-aca0db0fba42-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-bbfa07-qgnvt\" (UID: \"0de01d4b-467d-4d84-8f5d-aca0db0fba42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bbfa07-qgnvt" Oct 30 20:36:07.806335 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:07.806312 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0de01d4b-467d-4d84-8f5d-aca0db0fba42-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-bbfa07-qgnvt\" (UID: \"0de01d4b-467d-4d84-8f5d-aca0db0fba42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bbfa07-qgnvt" Oct 30 20:36:07.869288 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:07.869231 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxk2f\" (UniqueName: \"kubernetes.io/projected/0de01d4b-467d-4d84-8f5d-aca0db0fba42-kube-api-access-wxk2f\") pod \"maas-api-revoke-keys-e2e-worker-w3-bbfa07-qgnvt\" (UID: \"0de01d4b-467d-4d84-8f5d-aca0db0fba42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bbfa07-qgnvt" Oct 30 20:36:07.914275 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:07.914252 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bbfa07-qgnvt" Oct 30 20:36:08.282259 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:36:08.282234 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0de01d4b_467d_4d84_8f5d_aca0db0fba42.slice/crio-aec47b26ea271ad012649edaaab48e740f8f387802f4acbb4b9e80269a5723fe WatchSource:0}: Error finding container aec47b26ea271ad012649edaaab48e740f8f387802f4acbb4b9e80269a5723fe: Status 404 returned error can't find the container with id aec47b26ea271ad012649edaaab48e740f8f387802f4acbb4b9e80269a5723fe Oct 30 20:36:08.284806 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:08.284785 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bbfa07-qgnvt"] Oct 30 20:36:08.483901 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:08.483867 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bbfa07-qgnvt" event={"ID":"0de01d4b-467d-4d84-8f5d-aca0db0fba42","Type":"ContainerStarted","Data":"aec47b26ea271ad012649edaaab48e740f8f387802f4acbb4b9e80269a5723fe"} Oct 30 20:36:09.490866 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:09.490770 4292 generic.go:356] "Generic (PLEG): container finished" podID="0de01d4b-467d-4d84-8f5d-aca0db0fba42" containerID="244537e2840f179f396186fe7db08ed4fdacf335c1701b0cd0b2a534cff4af95" exitCode=0 Oct 30 20:36:09.491283 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:09.490862 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bbfa07-qgnvt" event={"ID":"0de01d4b-467d-4d84-8f5d-aca0db0fba42","Type":"ContainerDied","Data":"244537e2840f179f396186fe7db08ed4fdacf335c1701b0cd0b2a534cff4af95"} Oct 30 20:36:10.625735 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:10.625712 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bbfa07-qgnvt" Oct 30 20:36:10.735025 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:10.734991 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0de01d4b-467d-4d84-8f5d-aca0db0fba42-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0de01d4b-467d-4d84-8f5d-aca0db0fba42-maas-api-openshift-ca\") pod \"0de01d4b-467d-4d84-8f5d-aca0db0fba42\" (UID: \"0de01d4b-467d-4d84-8f5d-aca0db0fba42\") " Oct 30 20:36:10.735025 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:10.735028 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0de01d4b-467d-4d84-8f5d-aca0db0fba42-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0de01d4b-467d-4d84-8f5d-aca0db0fba42-maas-api-xks-ca\") pod \"0de01d4b-467d-4d84-8f5d-aca0db0fba42\" (UID: \"0de01d4b-467d-4d84-8f5d-aca0db0fba42\") " Oct 30 20:36:10.735239 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:10.735077 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0de01d4b-467d-4d84-8f5d-aca0db0fba42-kube-api-access-wxk2f\" (UniqueName: \"kubernetes.io/projected/0de01d4b-467d-4d84-8f5d-aca0db0fba42-kube-api-access-wxk2f\") pod \"0de01d4b-467d-4d84-8f5d-aca0db0fba42\" (UID: \"0de01d4b-467d-4d84-8f5d-aca0db0fba42\") " Oct 30 20:36:10.735409 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:10.735384 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0de01d4b-467d-4d84-8f5d-aca0db0fba42-maas-api-openshift-ca" pod "0de01d4b-467d-4d84-8f5d-aca0db0fba42" (UID: "0de01d4b-467d-4d84-8f5d-aca0db0fba42"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:36:10.735573 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:10.735550 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0de01d4b-467d-4d84-8f5d-aca0db0fba42-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:36:10.737244 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:10.737222 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0de01d4b-467d-4d84-8f5d-aca0db0fba42-kube-api-access-wxk2f" pod "0de01d4b-467d-4d84-8f5d-aca0db0fba42" (UID: "0de01d4b-467d-4d84-8f5d-aca0db0fba42"). InnerVolumeSpecName "kube-api-access-wxk2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:36:10.737526 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:10.737468 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0de01d4b-467d-4d84-8f5d-aca0db0fba42-maas-api-xks-ca" pod "0de01d4b-467d-4d84-8f5d-aca0db0fba42" (UID: "0de01d4b-467d-4d84-8f5d-aca0db0fba42"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:36:10.836547 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:10.836522 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0de01d4b-467d-4d84-8f5d-aca0db0fba42-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:36:10.836547 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:10.836545 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxk2f\" (UniqueName: \"kubernetes.io/projected/0de01d4b-467d-4d84-8f5d-aca0db0fba42-kube-api-access-wxk2f\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:36:11.502661 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:11.502626 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bbfa07-qgnvt" event={"ID":"0de01d4b-467d-4d84-8f5d-aca0db0fba42","Type":"ContainerDied","Data":"aec47b26ea271ad012649edaaab48e740f8f387802f4acbb4b9e80269a5723fe"} Oct 30 20:36:11.502661 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:11.502659 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bbfa07-qgnvt" Oct 30 20:36:11.502883 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:11.502681 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aec47b26ea271ad012649edaaab48e740f8f387802f4acbb4b9e80269a5723fe" Oct 30 20:36:12.009828 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:12.009792 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-b37684-zxwlt"] Oct 30 20:36:12.017765 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:12.017738 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-b37684-zxwlt"] Oct 30 20:36:12.060378 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:12.060338 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-bbfa07/distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2"] Oct 30 20:36:12.062751 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:12.062733 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-bbfa07/distinct-w3-bbfa07-kserve-67bb47494b-t8t7l"] Oct 30 20:36:12.079367 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:12.079327 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-bbfa07/premium-w3-bbfa07-kserve-5675444c9-x42d7"] Oct 30 20:36:12.582158 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:12.582126 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d4b6b54-6d2f-4ffa-9586-e80b29c3295d" path="/var/lib/kubelet/pods/5d4b6b54-6d2f-4ffa-9586-e80b29c3295d/volumes" Oct 30 20:36:13.220938 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:36:13.220898 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:36:19.134909 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:19.134877 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-bbfa07-5d784fb568-6ddcf"] Oct 30 20:36:19.242692 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:19.242657 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-bbfa07-658b9f6ff5-qxnqd"] Oct 30 20:36:21.148699 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:21.148666 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-98a19b-h29wq"] Oct 30 20:36:21.149271 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:21.149252 4292 cpu_manager.go:414] "containerMap: removing container" podUID="0de01d4b-467d-4d84-8f5d-aca0db0fba42" containerName="revoke-keys" Oct 30 20:36:21.149393 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:21.149272 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0de01d4b-467d-4d84-8f5d-aca0db0fba42" containerName="revoke-keys" Oct 30 20:36:21.149481 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:21.149466 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="0de01d4b-467d-4d84-8f5d-aca0db0fba42" containerName="revoke-keys" Oct 30 20:36:21.152998 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:21.152978 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-98a19b-h29wq" Oct 30 20:36:21.208248 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:21.208215 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-98a19b-h29wq"] Oct 30 20:36:21.255928 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:21.255899 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:36:21.256088 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:21.255938 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\"" Oct 30 20:36:21.268937 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:21.268890 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3165f4d2-0fa3-4358-8204-d1e5b46e5d41-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-98a19b-h29wq\" (UID: \"3165f4d2-0fa3-4358-8204-d1e5b46e5d41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-98a19b-h29wq" Oct 30 20:36:21.269106 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:21.269005 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3165f4d2-0fa3-4358-8204-d1e5b46e5d41-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-98a19b-h29wq\" (UID: \"3165f4d2-0fa3-4358-8204-d1e5b46e5d41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-98a19b-h29wq" Oct 30 20:36:21.269163 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:21.269150 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsg4g\" (UniqueName: \"kubernetes.io/projected/3165f4d2-0fa3-4358-8204-d1e5b46e5d41-kube-api-access-qsg4g\") pod \"maas-api-revoke-keys-e2e-worker-w1-98a19b-h29wq\" (UID: \"3165f4d2-0fa3-4358-8204-d1e5b46e5d41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-98a19b-h29wq" Oct 30 20:36:21.370652 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:21.370617 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsg4g\" (UniqueName: \"kubernetes.io/projected/3165f4d2-0fa3-4358-8204-d1e5b46e5d41-kube-api-access-qsg4g\") pod \"maas-api-revoke-keys-e2e-worker-w1-98a19b-h29wq\" (UID: \"3165f4d2-0fa3-4358-8204-d1e5b46e5d41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-98a19b-h29wq" Oct 30 20:36:21.370825 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:21.370678 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3165f4d2-0fa3-4358-8204-d1e5b46e5d41-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-98a19b-h29wq\" (UID: \"3165f4d2-0fa3-4358-8204-d1e5b46e5d41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-98a19b-h29wq" Oct 30 20:36:21.370825 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:21.370706 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3165f4d2-0fa3-4358-8204-d1e5b46e5d41-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-98a19b-h29wq\" (UID: \"3165f4d2-0fa3-4358-8204-d1e5b46e5d41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-98a19b-h29wq" Oct 30 20:36:21.371380 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:21.371344 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3165f4d2-0fa3-4358-8204-d1e5b46e5d41-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-98a19b-h29wq\" (UID: \"3165f4d2-0fa3-4358-8204-d1e5b46e5d41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-98a19b-h29wq" Oct 30 20:36:21.373011 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:21.372987 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3165f4d2-0fa3-4358-8204-d1e5b46e5d41-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-98a19b-h29wq\" (UID: \"3165f4d2-0fa3-4358-8204-d1e5b46e5d41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-98a19b-h29wq" Oct 30 20:36:21.433322 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:21.433249 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsg4g\" (UniqueName: \"kubernetes.io/projected/3165f4d2-0fa3-4358-8204-d1e5b46e5d41-kube-api-access-qsg4g\") pod \"maas-api-revoke-keys-e2e-worker-w1-98a19b-h29wq\" (UID: \"3165f4d2-0fa3-4358-8204-d1e5b46e5d41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-98a19b-h29wq" Oct 30 20:36:21.465720 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:21.465695 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-98a19b-h29wq" Oct 30 20:36:21.853974 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:36:21.853944 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3165f4d2_0fa3_4358_8204_d1e5b46e5d41.slice/crio-6e9f15e1f444dd43ba0f816ddb1e8bc510baeed6b2ad9f012fd82ee7eabfdcc3 WatchSource:0}: Error finding container 6e9f15e1f444dd43ba0f816ddb1e8bc510baeed6b2ad9f012fd82ee7eabfdcc3: Status 404 returned error can't find the container with id 6e9f15e1f444dd43ba0f816ddb1e8bc510baeed6b2ad9f012fd82ee7eabfdcc3 Oct 30 20:36:21.856447 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:21.856424 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-98a19b-h29wq"] Oct 30 20:36:22.593241 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:22.593209 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-98a19b-h29wq" event={"ID":"3165f4d2-0fa3-4358-8204-d1e5b46e5d41","Type":"ContainerStarted","Data":"fde6c50b57d144f647ed77f71672111dc42f5f62b8b27e5a50710a4b547af5aa"} Oct 30 20:36:22.593703 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:22.593255 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-98a19b-h29wq" event={"ID":"3165f4d2-0fa3-4358-8204-d1e5b46e5d41","Type":"ContainerStarted","Data":"6e9f15e1f444dd43ba0f816ddb1e8bc510baeed6b2ad9f012fd82ee7eabfdcc3"} Oct 30 20:36:22.712504 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:22.712448 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-98a19b-h29wq" podStartSLOduration=1.7124361559999999 podStartE2EDuration="1.712436156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:36:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:36:22.711718461 +0000 UTC m=+4286.230033923" watchObservedRunningTime="2026-10-30 20:36:22.712436156 +0000 UTC m=+4286.230751612" Oct 30 20:36:23.604294 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:23.604251 4292 generic.go:356] "Generic (PLEG): container finished" podID="3165f4d2-0fa3-4358-8204-d1e5b46e5d41" containerID="fde6c50b57d144f647ed77f71672111dc42f5f62b8b27e5a50710a4b547af5aa" exitCode=0 Oct 30 20:36:23.604750 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:23.604322 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-98a19b-h29wq" event={"ID":"3165f4d2-0fa3-4358-8204-d1e5b46e5d41","Type":"ContainerDied","Data":"fde6c50b57d144f647ed77f71672111dc42f5f62b8b27e5a50710a4b547af5aa"} Oct 30 20:36:24.135773 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:24.135724 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-bbfa07-5d784fb568-6ddcf" podUID="1f356d7e-7b1a-4b51-a36f-a809a4a4f91f" containerName="payload-processing" containerID="cri-o://2cffd75e0dbf07dd8029601824695fe3f485818d6b7e0f463051d032b36930e7" gracePeriod=25 Oct 30 20:36:24.243876 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:24.243807 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bbfa07-658b9f6ff5-qxnqd" podUID="d0946fd9-55cf-4c53-90af-22020e7b41ac" containerName="payload-pre-processing" containerID="cri-o://ca77cb60489d7c4ab5a0997b1863d00a80ca659ce6d7f37b9a877f6295747064" gracePeriod=25 Oct 30 20:36:24.767581 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:24.767557 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-98a19b-h29wq" Oct 30 20:36:24.914046 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:24.913945 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3165f4d2-0fa3-4358-8204-d1e5b46e5d41-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3165f4d2-0fa3-4358-8204-d1e5b46e5d41-maas-api-xks-ca\") pod \"3165f4d2-0fa3-4358-8204-d1e5b46e5d41\" (UID: \"3165f4d2-0fa3-4358-8204-d1e5b46e5d41\") " Oct 30 20:36:24.914046 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:24.913995 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3165f4d2-0fa3-4358-8204-d1e5b46e5d41-kube-api-access-qsg4g\" (UniqueName: \"kubernetes.io/projected/3165f4d2-0fa3-4358-8204-d1e5b46e5d41-kube-api-access-qsg4g\") pod \"3165f4d2-0fa3-4358-8204-d1e5b46e5d41\" (UID: \"3165f4d2-0fa3-4358-8204-d1e5b46e5d41\") " Oct 30 20:36:24.914303 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:24.914115 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3165f4d2-0fa3-4358-8204-d1e5b46e5d41-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3165f4d2-0fa3-4358-8204-d1e5b46e5d41-maas-api-openshift-ca\") pod \"3165f4d2-0fa3-4358-8204-d1e5b46e5d41\" (UID: \"3165f4d2-0fa3-4358-8204-d1e5b46e5d41\") " Oct 30 20:36:24.914538 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:24.914511 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3165f4d2-0fa3-4358-8204-d1e5b46e5d41-maas-api-openshift-ca" pod "3165f4d2-0fa3-4358-8204-d1e5b46e5d41" (UID: "3165f4d2-0fa3-4358-8204-d1e5b46e5d41"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:36:24.917234 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:24.916345 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3165f4d2-0fa3-4358-8204-d1e5b46e5d41-maas-api-xks-ca" pod "3165f4d2-0fa3-4358-8204-d1e5b46e5d41" (UID: "3165f4d2-0fa3-4358-8204-d1e5b46e5d41"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:36:24.917234 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:24.916492 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3165f4d2-0fa3-4358-8204-d1e5b46e5d41-kube-api-access-qsg4g" pod "3165f4d2-0fa3-4358-8204-d1e5b46e5d41" (UID: "3165f4d2-0fa3-4358-8204-d1e5b46e5d41"). InnerVolumeSpecName "kube-api-access-qsg4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:36:25.015962 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:25.015928 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3165f4d2-0fa3-4358-8204-d1e5b46e5d41-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:36:25.015962 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:25.015956 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3165f4d2-0fa3-4358-8204-d1e5b46e5d41-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:36:25.015962 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:25.015968 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsg4g\" (UniqueName: \"kubernetes.io/projected/3165f4d2-0fa3-4358-8204-d1e5b46e5d41-kube-api-access-qsg4g\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:36:25.607420 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:25.607384 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-98a19b/simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r"] Oct 30 20:36:25.617297 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:25.617274 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-98a19b-h29wq" Oct 30 20:36:25.617297 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:25.617298 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-98a19b-h29wq" event={"ID":"3165f4d2-0fa3-4358-8204-d1e5b46e5d41","Type":"ContainerDied","Data":"6e9f15e1f444dd43ba0f816ddb1e8bc510baeed6b2ad9f012fd82ee7eabfdcc3"} Oct 30 20:36:25.617551 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:25.617387 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e9f15e1f444dd43ba0f816ddb1e8bc510baeed6b2ad9f012fd82ee7eabfdcc3" Oct 30 20:36:27.082607 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.082573 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-bbfa07/distinct-w3-bbfa07-kserve-67bb47494b-t8t7l" podUID="16def41a-a612-4a4c-ada9-bb170b066af2" containerName="main" containerID="cri-o://dcaf4828ece39711e3074a02f18ffcf0c67c200e3c00ddce8b15ee87b8652547" gracePeriod=45 Oct 30 20:36:27.098250 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.098219 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-bbfa07/premium-w3-bbfa07-kserve-5675444c9-x42d7" podUID="e0b75746-c06c-407d-8f61-c795c519aad6" containerName="main" containerID="cri-o://3d73da7a92814ad72ece1fe671bfec46d0335b3e7f2c12d766e01be7ec48c87e" gracePeriod=45 Oct 30 20:36:27.351797 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.351771 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bbfa07/distinct-w3-bbfa07-kserve-67bb47494b-t8t7l" Oct 30 20:36:27.386745 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.386722 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bbfa07/premium-w3-bbfa07-kserve-5675444c9-x42d7" Oct 30 20:36:27.440818 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.440789 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16def41a-a612-4a4c-ada9-bb170b066af2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16def41a-a612-4a4c-ada9-bb170b066af2-tmp-dir\") pod \"16def41a-a612-4a4c-ada9-bb170b066af2\" (UID: \"16def41a-a612-4a4c-ada9-bb170b066af2\") " Oct 30 20:36:27.440985 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.440858 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16def41a-a612-4a4c-ada9-bb170b066af2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16def41a-a612-4a4c-ada9-bb170b066af2-model-cache\") pod \"16def41a-a612-4a4c-ada9-bb170b066af2\" (UID: \"16def41a-a612-4a4c-ada9-bb170b066af2\") " Oct 30 20:36:27.440985 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.440881 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16def41a-a612-4a4c-ada9-bb170b066af2-tls-certs\" (UniqueName: \"kubernetes.io/secret/16def41a-a612-4a4c-ada9-bb170b066af2-tls-certs\") pod \"16def41a-a612-4a4c-ada9-bb170b066af2\" (UID: \"16def41a-a612-4a4c-ada9-bb170b066af2\") " Oct 30 20:36:27.440985 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.440906 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16def41a-a612-4a4c-ada9-bb170b066af2-home\" (UniqueName: \"kubernetes.io/empty-dir/16def41a-a612-4a4c-ada9-bb170b066af2-home\") pod \"16def41a-a612-4a4c-ada9-bb170b066af2\" (UID: \"16def41a-a612-4a4c-ada9-bb170b066af2\") " Oct 30 20:36:27.440985 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.440944 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16def41a-a612-4a4c-ada9-bb170b066af2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/16def41a-a612-4a4c-ada9-bb170b066af2-dshm\") pod \"16def41a-a612-4a4c-ada9-bb170b066af2\" (UID: \"16def41a-a612-4a4c-ada9-bb170b066af2\") " Oct 30 20:36:27.440985 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.440973 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16def41a-a612-4a4c-ada9-bb170b066af2-kube-api-access-w2wlx\" (UniqueName: \"kubernetes.io/projected/16def41a-a612-4a4c-ada9-bb170b066af2-kube-api-access-w2wlx\") pod \"16def41a-a612-4a4c-ada9-bb170b066af2\" (UID: \"16def41a-a612-4a4c-ada9-bb170b066af2\") " Oct 30 20:36:27.441252 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.441098 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16def41a-a612-4a4c-ada9-bb170b066af2-tmp-dir" pod "16def41a-a612-4a4c-ada9-bb170b066af2" (UID: "16def41a-a612-4a4c-ada9-bb170b066af2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:36:27.441252 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.441195 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16def41a-a612-4a4c-ada9-bb170b066af2-home" pod "16def41a-a612-4a4c-ada9-bb170b066af2" (UID: "16def41a-a612-4a4c-ada9-bb170b066af2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:36:27.441252 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.441220 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16def41a-a612-4a4c-ada9-bb170b066af2-model-cache" pod "16def41a-a612-4a4c-ada9-bb170b066af2" (UID: "16def41a-a612-4a4c-ada9-bb170b066af2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:36:27.441454 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.441426 4292 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16def41a-a612-4a4c-ada9-bb170b066af2-tmp-dir\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:36:27.441454 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.441455 4292 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16def41a-a612-4a4c-ada9-bb170b066af2-model-cache\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:36:27.441648 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.441470 4292 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16def41a-a612-4a4c-ada9-bb170b066af2-home\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:36:27.443231 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.443209 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16def41a-a612-4a4c-ada9-bb170b066af2-dshm" pod "16def41a-a612-4a4c-ada9-bb170b066af2" (UID: "16def41a-a612-4a4c-ada9-bb170b066af2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:36:27.443320 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.443254 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16def41a-a612-4a4c-ada9-bb170b066af2-kube-api-access-w2wlx" pod "16def41a-a612-4a4c-ada9-bb170b066af2" (UID: "16def41a-a612-4a4c-ada9-bb170b066af2"). InnerVolumeSpecName "kube-api-access-w2wlx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:36:27.443320 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.443262 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16def41a-a612-4a4c-ada9-bb170b066af2-tls-certs" pod "16def41a-a612-4a4c-ada9-bb170b066af2" (UID: "16def41a-a612-4a4c-ada9-bb170b066af2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:36:27.527478 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.527432 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-bbfa07/distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2" podUID="f562f81a-ebd3-4efa-889d-bf4a757527fe" containerName="main" containerID="cri-o://e3959c82e66eb88d22e421cd2eb52089348bee2b84576f26f7e5b87c20862b59" gracePeriod=45 Oct 30 20:36:27.542504 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.542480 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0b75746-c06c-407d-8f61-c795c519aad6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0b75746-c06c-407d-8f61-c795c519aad6-dshm\") pod \"e0b75746-c06c-407d-8f61-c795c519aad6\" (UID: \"e0b75746-c06c-407d-8f61-c795c519aad6\") " Oct 30 20:36:27.542650 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.542531 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0b75746-c06c-407d-8f61-c795c519aad6-kube-api-access-knc5n\" (UniqueName: \"kubernetes.io/projected/e0b75746-c06c-407d-8f61-c795c519aad6-kube-api-access-knc5n\") pod \"e0b75746-c06c-407d-8f61-c795c519aad6\" (UID: \"e0b75746-c06c-407d-8f61-c795c519aad6\") " Oct 30 20:36:27.542650 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.542572 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0b75746-c06c-407d-8f61-c795c519aad6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0b75746-c06c-407d-8f61-c795c519aad6-tmp-dir\") pod \"e0b75746-c06c-407d-8f61-c795c519aad6\" (UID: \"e0b75746-c06c-407d-8f61-c795c519aad6\") " Oct 30 20:36:27.542774 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.542652 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0b75746-c06c-407d-8f61-c795c519aad6-tls-certs\" (UniqueName: \"kubernetes.io/secret/e0b75746-c06c-407d-8f61-c795c519aad6-tls-certs\") pod \"e0b75746-c06c-407d-8f61-c795c519aad6\" (UID: \"e0b75746-c06c-407d-8f61-c795c519aad6\") " Oct 30 20:36:27.542774 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.542722 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0b75746-c06c-407d-8f61-c795c519aad6-home\" (UniqueName: \"kubernetes.io/empty-dir/e0b75746-c06c-407d-8f61-c795c519aad6-home\") pod \"e0b75746-c06c-407d-8f61-c795c519aad6\" (UID: \"e0b75746-c06c-407d-8f61-c795c519aad6\") " Oct 30 20:36:27.542878 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.542799 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0b75746-c06c-407d-8f61-c795c519aad6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0b75746-c06c-407d-8f61-c795c519aad6-model-cache\") pod \"e0b75746-c06c-407d-8f61-c795c519aad6\" (UID: \"e0b75746-c06c-407d-8f61-c795c519aad6\") " Oct 30 20:36:27.542933 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.542905 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0b75746-c06c-407d-8f61-c795c519aad6-tmp-dir" pod "e0b75746-c06c-407d-8f61-c795c519aad6" (UID: "e0b75746-c06c-407d-8f61-c795c519aad6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:36:27.543150 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.543124 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0b75746-c06c-407d-8f61-c795c519aad6-home" pod "e0b75746-c06c-407d-8f61-c795c519aad6" (UID: "e0b75746-c06c-407d-8f61-c795c519aad6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:36:27.543260 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.543132 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0b75746-c06c-407d-8f61-c795c519aad6-model-cache" pod "e0b75746-c06c-407d-8f61-c795c519aad6" (UID: "e0b75746-c06c-407d-8f61-c795c519aad6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:36:27.543322 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.543284 4292 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16def41a-a612-4a4c-ada9-bb170b066af2-tls-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:36:27.543404 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.543301 4292 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0b75746-c06c-407d-8f61-c795c519aad6-home\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:36:27.543404 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.543397 4292 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16def41a-a612-4a4c-ada9-bb170b066af2-dshm\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:36:27.543527 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.543412 4292 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0b75746-c06c-407d-8f61-c795c519aad6-model-cache\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:36:27.543527 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.543429 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2wlx\" (UniqueName: \"kubernetes.io/projected/16def41a-a612-4a4c-ada9-bb170b066af2-kube-api-access-w2wlx\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:36:27.543527 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.543445 4292 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0b75746-c06c-407d-8f61-c795c519aad6-tmp-dir\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:36:27.544826 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.544802 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0b75746-c06c-407d-8f61-c795c519aad6-tls-certs" pod "e0b75746-c06c-407d-8f61-c795c519aad6" (UID: "e0b75746-c06c-407d-8f61-c795c519aad6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:36:27.544913 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.544870 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0b75746-c06c-407d-8f61-c795c519aad6-dshm" pod "e0b75746-c06c-407d-8f61-c795c519aad6" (UID: "e0b75746-c06c-407d-8f61-c795c519aad6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:36:27.545287 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.545265 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0b75746-c06c-407d-8f61-c795c519aad6-kube-api-access-knc5n" pod "e0b75746-c06c-407d-8f61-c795c519aad6" (UID: "e0b75746-c06c-407d-8f61-c795c519aad6"). InnerVolumeSpecName "kube-api-access-knc5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:36:27.563477 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.563448 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bbfa07/distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2" podUID="f562f81a-ebd3-4efa-889d-bf4a757527fe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": dial tcp 10.133.0.60:8000: connect: connection refused" Oct 30 20:36:27.631018 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.630929 4292 generic.go:356] "Generic (PLEG): container finished" podID="16def41a-a612-4a4c-ada9-bb170b066af2" containerID="dcaf4828ece39711e3074a02f18ffcf0c67c200e3c00ddce8b15ee87b8652547" exitCode=0 Oct 30 20:36:27.631018 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.630984 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bbfa07/distinct-w3-bbfa07-kserve-67bb47494b-t8t7l" Oct 30 20:36:27.631243 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.631014 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bbfa07/distinct-w3-bbfa07-kserve-67bb47494b-t8t7l" event={"ID":"16def41a-a612-4a4c-ada9-bb170b066af2","Type":"ContainerDied","Data":"dcaf4828ece39711e3074a02f18ffcf0c67c200e3c00ddce8b15ee87b8652547"} Oct 30 20:36:27.631243 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.631079 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bbfa07/distinct-w3-bbfa07-kserve-67bb47494b-t8t7l" event={"ID":"16def41a-a612-4a4c-ada9-bb170b066af2","Type":"ContainerDied","Data":"70ae3e1d114c53f8b562bff44ca512184fbb822f1da354c456f5f24c4ce7c2b4"} Oct 30 20:36:27.631243 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.631103 4292 scope.go:122] "RemoveContainer" containerID="dcaf4828ece39711e3074a02f18ffcf0c67c200e3c00ddce8b15ee87b8652547" Oct 30 20:36:27.632770 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.632736 4292 generic.go:356] "Generic (PLEG): container finished" podID="e0b75746-c06c-407d-8f61-c795c519aad6" containerID="3d73da7a92814ad72ece1fe671bfec46d0335b3e7f2c12d766e01be7ec48c87e" exitCode=0 Oct 30 20:36:27.632886 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.632790 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bbfa07/premium-w3-bbfa07-kserve-5675444c9-x42d7" Oct 30 20:36:27.632956 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.632900 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bbfa07/premium-w3-bbfa07-kserve-5675444c9-x42d7" event={"ID":"e0b75746-c06c-407d-8f61-c795c519aad6","Type":"ContainerDied","Data":"3d73da7a92814ad72ece1fe671bfec46d0335b3e7f2c12d766e01be7ec48c87e"} Oct 30 20:36:27.633027 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.632959 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bbfa07/premium-w3-bbfa07-kserve-5675444c9-x42d7" event={"ID":"e0b75746-c06c-407d-8f61-c795c519aad6","Type":"ContainerDied","Data":"5a666776a564544573674eb2e64b06143d021eb10006622e7378cb3b00dc3c14"} Oct 30 20:36:27.641641 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.641269 4292 scope.go:122] "RemoveContainer" containerID="dcaf4828ece39711e3074a02f18ffcf0c67c200e3c00ddce8b15ee87b8652547" Oct 30 20:36:27.642079 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:36:27.642049 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcaf4828ece39711e3074a02f18ffcf0c67c200e3c00ddce8b15ee87b8652547\": container with ID starting with dcaf4828ece39711e3074a02f18ffcf0c67c200e3c00ddce8b15ee87b8652547 not found: ID does not exist" containerID="dcaf4828ece39711e3074a02f18ffcf0c67c200e3c00ddce8b15ee87b8652547" Oct 30 20:36:27.642230 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.642088 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcaf4828ece39711e3074a02f18ffcf0c67c200e3c00ddce8b15ee87b8652547"} err="failed to get container status \"dcaf4828ece39711e3074a02f18ffcf0c67c200e3c00ddce8b15ee87b8652547\": rpc error: code = NotFound desc = could not find container \"dcaf4828ece39711e3074a02f18ffcf0c67c200e3c00ddce8b15ee87b8652547\": container with ID starting with dcaf4828ece39711e3074a02f18ffcf0c67c200e3c00ddce8b15ee87b8652547 not found: ID does not exist" Oct 30 20:36:27.642230 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.642112 4292 scope.go:122] "RemoveContainer" containerID="3d73da7a92814ad72ece1fe671bfec46d0335b3e7f2c12d766e01be7ec48c87e" Oct 30 20:36:27.645020 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.644997 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knc5n\" (UniqueName: \"kubernetes.io/projected/e0b75746-c06c-407d-8f61-c795c519aad6-kube-api-access-knc5n\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:36:27.645081 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.645029 4292 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0b75746-c06c-407d-8f61-c795c519aad6-tls-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:36:27.645081 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.645044 4292 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0b75746-c06c-407d-8f61-c795c519aad6-dshm\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:36:27.651841 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.651764 4292 scope.go:122] "RemoveContainer" containerID="3d73da7a92814ad72ece1fe671bfec46d0335b3e7f2c12d766e01be7ec48c87e" Oct 30 20:36:27.652278 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:36:27.652254 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d73da7a92814ad72ece1fe671bfec46d0335b3e7f2c12d766e01be7ec48c87e\": container with ID starting with 3d73da7a92814ad72ece1fe671bfec46d0335b3e7f2c12d766e01be7ec48c87e not found: ID does not exist" containerID="3d73da7a92814ad72ece1fe671bfec46d0335b3e7f2c12d766e01be7ec48c87e" Oct 30 20:36:27.652392 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.652286 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d73da7a92814ad72ece1fe671bfec46d0335b3e7f2c12d766e01be7ec48c87e"} err="failed to get container status \"3d73da7a92814ad72ece1fe671bfec46d0335b3e7f2c12d766e01be7ec48c87e\": rpc error: code = NotFound desc = could not find container \"3d73da7a92814ad72ece1fe671bfec46d0335b3e7f2c12d766e01be7ec48c87e\": container with ID starting with 3d73da7a92814ad72ece1fe671bfec46d0335b3e7f2c12d766e01be7ec48c87e not found: ID does not exist" Oct 30 20:36:27.767966 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.767940 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bbfa07/distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2" Oct 30 20:36:27.800536 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.800511 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-bbfa07/premium-w3-bbfa07-kserve-5675444c9-x42d7"] Oct 30 20:36:27.806111 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.806083 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-bbfa07/premium-w3-bbfa07-kserve-5675444c9-x42d7"] Oct 30 20:36:27.846056 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.846033 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f562f81a-ebd3-4efa-889d-bf4a757527fe-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f562f81a-ebd3-4efa-889d-bf4a757527fe-dshm\") pod \"f562f81a-ebd3-4efa-889d-bf4a757527fe\" (UID: \"f562f81a-ebd3-4efa-889d-bf4a757527fe\") " Oct 30 20:36:27.846198 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.846140 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f562f81a-ebd3-4efa-889d-bf4a757527fe-kube-api-access-bpbmq\" (UniqueName: \"kubernetes.io/projected/f562f81a-ebd3-4efa-889d-bf4a757527fe-kube-api-access-bpbmq\") pod \"f562f81a-ebd3-4efa-889d-bf4a757527fe\" (UID: \"f562f81a-ebd3-4efa-889d-bf4a757527fe\") " Oct 30 20:36:27.846252 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.846209 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f562f81a-ebd3-4efa-889d-bf4a757527fe-home\" (UniqueName: \"kubernetes.io/empty-dir/f562f81a-ebd3-4efa-889d-bf4a757527fe-home\") pod \"f562f81a-ebd3-4efa-889d-bf4a757527fe\" (UID: \"f562f81a-ebd3-4efa-889d-bf4a757527fe\") " Oct 30 20:36:27.846320 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.846249 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f562f81a-ebd3-4efa-889d-bf4a757527fe-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f562f81a-ebd3-4efa-889d-bf4a757527fe-tmp-dir\") pod \"f562f81a-ebd3-4efa-889d-bf4a757527fe\" (UID: \"f562f81a-ebd3-4efa-889d-bf4a757527fe\") " Oct 30 20:36:27.846320 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.846286 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f562f81a-ebd3-4efa-889d-bf4a757527fe-tls-certs\" (UniqueName: \"kubernetes.io/secret/f562f81a-ebd3-4efa-889d-bf4a757527fe-tls-certs\") pod \"f562f81a-ebd3-4efa-889d-bf4a757527fe\" (UID: \"f562f81a-ebd3-4efa-889d-bf4a757527fe\") " Oct 30 20:36:27.846466 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.846368 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f562f81a-ebd3-4efa-889d-bf4a757527fe-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f562f81a-ebd3-4efa-889d-bf4a757527fe-model-cache\") pod \"f562f81a-ebd3-4efa-889d-bf4a757527fe\" (UID: \"f562f81a-ebd3-4efa-889d-bf4a757527fe\") " Oct 30 20:36:27.846530 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.846500 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f562f81a-ebd3-4efa-889d-bf4a757527fe-home" pod "f562f81a-ebd3-4efa-889d-bf4a757527fe" (UID: "f562f81a-ebd3-4efa-889d-bf4a757527fe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:36:27.846610 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.846565 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f562f81a-ebd3-4efa-889d-bf4a757527fe-tmp-dir" pod "f562f81a-ebd3-4efa-889d-bf4a757527fe" (UID: "f562f81a-ebd3-4efa-889d-bf4a757527fe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:36:27.846673 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.846655 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f562f81a-ebd3-4efa-889d-bf4a757527fe-model-cache" pod "f562f81a-ebd3-4efa-889d-bf4a757527fe" (UID: "f562f81a-ebd3-4efa-889d-bf4a757527fe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:36:27.846839 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.846819 4292 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f562f81a-ebd3-4efa-889d-bf4a757527fe-tmp-dir\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:36:27.846942 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.846843 4292 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f562f81a-ebd3-4efa-889d-bf4a757527fe-model-cache\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:36:27.846942 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.846857 4292 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f562f81a-ebd3-4efa-889d-bf4a757527fe-home\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:36:27.848443 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.848420 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f562f81a-ebd3-4efa-889d-bf4a757527fe-tls-certs" pod "f562f81a-ebd3-4efa-889d-bf4a757527fe" (UID: "f562f81a-ebd3-4efa-889d-bf4a757527fe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:36:27.848443 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.848428 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f562f81a-ebd3-4efa-889d-bf4a757527fe-dshm" pod "f562f81a-ebd3-4efa-889d-bf4a757527fe" (UID: "f562f81a-ebd3-4efa-889d-bf4a757527fe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:36:27.848560 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.848488 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f562f81a-ebd3-4efa-889d-bf4a757527fe-kube-api-access-bpbmq" pod "f562f81a-ebd3-4efa-889d-bf4a757527fe" (UID: "f562f81a-ebd3-4efa-889d-bf4a757527fe"). InnerVolumeSpecName "kube-api-access-bpbmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:36:27.918288 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.918213 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-bbfa07/distinct-w3-bbfa07-kserve-67bb47494b-t8t7l"] Oct 30 20:36:27.922390 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.922366 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-bbfa07/distinct-w3-bbfa07-kserve-67bb47494b-t8t7l"] Oct 30 20:36:27.947830 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.947796 4292 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f562f81a-ebd3-4efa-889d-bf4a757527fe-dshm\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:36:27.947830 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.947827 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpbmq\" (UniqueName: \"kubernetes.io/projected/f562f81a-ebd3-4efa-889d-bf4a757527fe-kube-api-access-bpbmq\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:36:27.948041 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:27.947842 4292 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f562f81a-ebd3-4efa-889d-bf4a757527fe-tls-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:36:28.581462 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:28.581426 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16def41a-a612-4a4c-ada9-bb170b066af2" path="/var/lib/kubelet/pods/16def41a-a612-4a4c-ada9-bb170b066af2/volumes" Oct 30 20:36:28.581876 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:28.581862 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0b75746-c06c-407d-8f61-c795c519aad6" path="/var/lib/kubelet/pods/e0b75746-c06c-407d-8f61-c795c519aad6/volumes" Oct 30 20:36:28.639537 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:28.639508 4292 generic.go:356] "Generic (PLEG): container finished" podID="f562f81a-ebd3-4efa-889d-bf4a757527fe" containerID="e3959c82e66eb88d22e421cd2eb52089348bee2b84576f26f7e5b87c20862b59" exitCode=0 Oct 30 20:36:28.639672 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:28.639572 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bbfa07/distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2" Oct 30 20:36:28.639672 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:28.639595 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bbfa07/distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2" event={"ID":"f562f81a-ebd3-4efa-889d-bf4a757527fe","Type":"ContainerDied","Data":"e3959c82e66eb88d22e421cd2eb52089348bee2b84576f26f7e5b87c20862b59"} Oct 30 20:36:28.639672 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:28.639632 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bbfa07/distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2" event={"ID":"f562f81a-ebd3-4efa-889d-bf4a757527fe","Type":"ContainerDied","Data":"112003ebbabc11c24b96769f7af9ea3d5a84f94d674fbe644ba38b75ab9bc68f"} Oct 30 20:36:28.639672 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:28.639653 4292 scope.go:122] "RemoveContainer" containerID="e3959c82e66eb88d22e421cd2eb52089348bee2b84576f26f7e5b87c20862b59" Oct 30 20:36:28.648361 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:28.648331 4292 scope.go:122] "RemoveContainer" containerID="e3959c82e66eb88d22e421cd2eb52089348bee2b84576f26f7e5b87c20862b59" Oct 30 20:36:28.648618 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:36:28.648598 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3959c82e66eb88d22e421cd2eb52089348bee2b84576f26f7e5b87c20862b59\": container with ID starting with e3959c82e66eb88d22e421cd2eb52089348bee2b84576f26f7e5b87c20862b59 not found: ID does not exist" containerID="e3959c82e66eb88d22e421cd2eb52089348bee2b84576f26f7e5b87c20862b59" Oct 30 20:36:28.648670 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:28.648631 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3959c82e66eb88d22e421cd2eb52089348bee2b84576f26f7e5b87c20862b59"} err="failed to get container status \"e3959c82e66eb88d22e421cd2eb52089348bee2b84576f26f7e5b87c20862b59\": rpc error: code = NotFound desc = could not find container \"e3959c82e66eb88d22e421cd2eb52089348bee2b84576f26f7e5b87c20862b59\": container with ID starting with e3959c82e66eb88d22e421cd2eb52089348bee2b84576f26f7e5b87c20862b59 not found: ID does not exist" Oct 30 20:36:28.807131 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:28.807096 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-bbfa07/distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2"] Oct 30 20:36:28.857874 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:28.857793 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-bbfa07/distinct-2-w3-bbfa07-kserve-5f948857d9-xbqw2"] Oct 30 20:36:30.582451 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:30.582411 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f562f81a-ebd3-4efa-889d-bf4a757527fe" path="/var/lib/kubelet/pods/f562f81a-ebd3-4efa-889d-bf4a757527fe/volumes" Oct 30 20:36:31.100107 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:31.100075 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-bbfa07-5d784fb568-6ddcf" podUID="1f356d7e-7b1a-4b51-a36f-a809a4a4f91f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:36:31.100255 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:31.100075 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bbfa07-658b9f6ff5-qxnqd" podUID="d0946fd9-55cf-4c53-90af-22020e7b41ac" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:36:35.075929 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:35.075895 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-98a19b-779f877d8-hgncx"] Oct 30 20:36:40.077126 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:40.077067 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-98a19b-779f877d8-hgncx" podUID="e8810ca2-97ba-4a61-84da-fc464be80c17" containerName="payload-processing" containerID="cri-o://bfb441cca7276ae3e08df2c77f3f4ec1a6c0a473e4347a1eec4ccee3f07a13b1" gracePeriod=25 Oct 30 20:36:40.632249 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:40.632176 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-98a19b/simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r" podUID="1f35e32e-1f09-413d-a364-14b8ac1019f2" containerName="main" containerID="cri-o://7d0089e0b6f5fde3277d4efb2b31ac19a0e1f76bdc2332548fe9b4293ddbb15b" gracePeriod=45 Oct 30 20:36:40.651980 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:40.651950 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-98a19b/simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r" podUID="1f35e32e-1f09-413d-a364-14b8ac1019f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": dial tcp 10.133.0.47:8000: connect: connection refused" Oct 30 20:36:40.883234 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:40.883178 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-98a19b/simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r" Oct 30 20:36:40.982128 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:40.982101 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f35e32e-1f09-413d-a364-14b8ac1019f2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f35e32e-1f09-413d-a364-14b8ac1019f2-model-cache\") pod \"1f35e32e-1f09-413d-a364-14b8ac1019f2\" (UID: \"1f35e32e-1f09-413d-a364-14b8ac1019f2\") " Oct 30 20:36:40.982324 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:40.982163 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f35e32e-1f09-413d-a364-14b8ac1019f2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f35e32e-1f09-413d-a364-14b8ac1019f2-tmp-dir\") pod \"1f35e32e-1f09-413d-a364-14b8ac1019f2\" (UID: \"1f35e32e-1f09-413d-a364-14b8ac1019f2\") " Oct 30 20:36:40.982324 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:40.982227 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f35e32e-1f09-413d-a364-14b8ac1019f2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f35e32e-1f09-413d-a364-14b8ac1019f2-dshm\") pod \"1f35e32e-1f09-413d-a364-14b8ac1019f2\" (UID: \"1f35e32e-1f09-413d-a364-14b8ac1019f2\") " Oct 30 20:36:40.982324 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:40.982246 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f35e32e-1f09-413d-a364-14b8ac1019f2-home\" (UniqueName: \"kubernetes.io/empty-dir/1f35e32e-1f09-413d-a364-14b8ac1019f2-home\") pod \"1f35e32e-1f09-413d-a364-14b8ac1019f2\" (UID: \"1f35e32e-1f09-413d-a364-14b8ac1019f2\") " Oct 30 20:36:40.982324 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:40.982292 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f35e32e-1f09-413d-a364-14b8ac1019f2-tls-certs\" (UniqueName: \"kubernetes.io/secret/1f35e32e-1f09-413d-a364-14b8ac1019f2-tls-certs\") pod \"1f35e32e-1f09-413d-a364-14b8ac1019f2\" (UID: \"1f35e32e-1f09-413d-a364-14b8ac1019f2\") " Oct 30 20:36:40.982584 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:40.982361 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f35e32e-1f09-413d-a364-14b8ac1019f2-kube-api-access-jdsph\" (UniqueName: \"kubernetes.io/projected/1f35e32e-1f09-413d-a364-14b8ac1019f2-kube-api-access-jdsph\") pod \"1f35e32e-1f09-413d-a364-14b8ac1019f2\" (UID: \"1f35e32e-1f09-413d-a364-14b8ac1019f2\") " Oct 30 20:36:40.982584 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:40.982389 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f35e32e-1f09-413d-a364-14b8ac1019f2-model-cache" pod "1f35e32e-1f09-413d-a364-14b8ac1019f2" (UID: "1f35e32e-1f09-413d-a364-14b8ac1019f2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:36:40.982584 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:40.982477 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f35e32e-1f09-413d-a364-14b8ac1019f2-tmp-dir" pod "1f35e32e-1f09-413d-a364-14b8ac1019f2" (UID: "1f35e32e-1f09-413d-a364-14b8ac1019f2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:36:40.982584 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:40.982556 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f35e32e-1f09-413d-a364-14b8ac1019f2-home" pod "1f35e32e-1f09-413d-a364-14b8ac1019f2" (UID: "1f35e32e-1f09-413d-a364-14b8ac1019f2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:36:40.982850 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:40.982829 4292 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f35e32e-1f09-413d-a364-14b8ac1019f2-model-cache\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:36:40.982850 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:40.982849 4292 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f35e32e-1f09-413d-a364-14b8ac1019f2-tmp-dir\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:36:40.983051 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:40.982860 4292 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f35e32e-1f09-413d-a364-14b8ac1019f2-home\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:36:40.984487 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:40.984460 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f35e32e-1f09-413d-a364-14b8ac1019f2-tls-certs" pod "1f35e32e-1f09-413d-a364-14b8ac1019f2" (UID: "1f35e32e-1f09-413d-a364-14b8ac1019f2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:36:40.984487 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:40.984470 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f35e32e-1f09-413d-a364-14b8ac1019f2-dshm" pod "1f35e32e-1f09-413d-a364-14b8ac1019f2" (UID: "1f35e32e-1f09-413d-a364-14b8ac1019f2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:36:40.984608 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:40.984547 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f35e32e-1f09-413d-a364-14b8ac1019f2-kube-api-access-jdsph" pod "1f35e32e-1f09-413d-a364-14b8ac1019f2" (UID: "1f35e32e-1f09-413d-a364-14b8ac1019f2"). InnerVolumeSpecName "kube-api-access-jdsph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:36:41.083606 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:41.083574 4292 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f35e32e-1f09-413d-a364-14b8ac1019f2-dshm\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:36:41.083606 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:41.083598 4292 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f35e32e-1f09-413d-a364-14b8ac1019f2-tls-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:36:41.083606 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:41.083610 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdsph\" (UniqueName: \"kubernetes.io/projected/1f35e32e-1f09-413d-a364-14b8ac1019f2-kube-api-access-jdsph\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:36:41.100345 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:41.100316 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-bbfa07-5d784fb568-6ddcf" podUID="1f356d7e-7b1a-4b51-a36f-a809a4a4f91f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:36:41.100345 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:41.100327 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bbfa07-658b9f6ff5-qxnqd" podUID="d0946fd9-55cf-4c53-90af-22020e7b41ac" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:36:41.144650 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:41.144565 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-98a19b-779f877d8-hgncx" podUID="e8810ca2-97ba-4a61-84da-fc464be80c17" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:36:41.724911 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:41.724872 4292 generic.go:356] "Generic (PLEG): container finished" podID="1f35e32e-1f09-413d-a364-14b8ac1019f2" containerID="7d0089e0b6f5fde3277d4efb2b31ac19a0e1f76bdc2332548fe9b4293ddbb15b" exitCode=0 Oct 30 20:36:41.725125 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:41.724935 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-98a19b/simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r" Oct 30 20:36:41.725125 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:41.724950 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-98a19b/simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r" event={"ID":"1f35e32e-1f09-413d-a364-14b8ac1019f2","Type":"ContainerDied","Data":"7d0089e0b6f5fde3277d4efb2b31ac19a0e1f76bdc2332548fe9b4293ddbb15b"} Oct 30 20:36:41.725125 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:41.725009 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-98a19b/simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r" event={"ID":"1f35e32e-1f09-413d-a364-14b8ac1019f2","Type":"ContainerDied","Data":"c441d1a5aba7925a7addd7bc28b36005309a58d63d73cddcb8dcb033b3973b9a"} Oct 30 20:36:41.725125 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:41.725032 4292 scope.go:122] "RemoveContainer" containerID="7d0089e0b6f5fde3277d4efb2b31ac19a0e1f76bdc2332548fe9b4293ddbb15b" Oct 30 20:36:41.734147 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:41.734129 4292 scope.go:122] "RemoveContainer" containerID="7d0089e0b6f5fde3277d4efb2b31ac19a0e1f76bdc2332548fe9b4293ddbb15b" Oct 30 20:36:41.734420 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:36:41.734395 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d0089e0b6f5fde3277d4efb2b31ac19a0e1f76bdc2332548fe9b4293ddbb15b\": container with ID starting with 7d0089e0b6f5fde3277d4efb2b31ac19a0e1f76bdc2332548fe9b4293ddbb15b not found: ID does not exist" containerID="7d0089e0b6f5fde3277d4efb2b31ac19a0e1f76bdc2332548fe9b4293ddbb15b" Oct 30 20:36:41.734539 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:41.734426 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d0089e0b6f5fde3277d4efb2b31ac19a0e1f76bdc2332548fe9b4293ddbb15b"} err="failed to get container status \"7d0089e0b6f5fde3277d4efb2b31ac19a0e1f76bdc2332548fe9b4293ddbb15b\": rpc error: code = NotFound desc = could not find container \"7d0089e0b6f5fde3277d4efb2b31ac19a0e1f76bdc2332548fe9b4293ddbb15b\": container with ID starting with 7d0089e0b6f5fde3277d4efb2b31ac19a0e1f76bdc2332548fe9b4293ddbb15b not found: ID does not exist" Oct 30 20:36:41.895281 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:41.895249 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-98a19b/simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r"] Oct 30 20:36:41.945773 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:41.945740 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-98a19b/simulator-w1-98a19b-kserve-6cd5b7fccb-bpb9r"] Oct 30 20:36:42.581444 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:42.581413 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f35e32e-1f09-413d-a364-14b8ac1019f2" path="/var/lib/kubelet/pods/1f35e32e-1f09-413d-a364-14b8ac1019f2/volumes" Oct 30 20:36:43.046529 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:43.046493 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a65f438e-ctj4f"] Oct 30 20:36:43.046982 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:43.046964 4292 cpu_manager.go:414] "containerMap: removing container" podUID="e0b75746-c06c-407d-8f61-c795c519aad6" containerName="main" Oct 30 20:36:43.047145 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:43.046985 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0b75746-c06c-407d-8f61-c795c519aad6" containerName="main" Oct 30 20:36:43.047145 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:43.047006 4292 cpu_manager.go:414] "containerMap: removing container" podUID="3165f4d2-0fa3-4358-8204-d1e5b46e5d41" containerName="revoke-keys" Oct 30 20:36:43.047145 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:43.047013 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3165f4d2-0fa3-4358-8204-d1e5b46e5d41" containerName="revoke-keys" Oct 30 20:36:43.047145 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:43.047026 4292 cpu_manager.go:414] "containerMap: removing container" podUID="16def41a-a612-4a4c-ada9-bb170b066af2" containerName="main" Oct 30 20:36:43.047145 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:43.047033 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16def41a-a612-4a4c-ada9-bb170b066af2" containerName="main" Oct 30 20:36:43.047145 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:43.047047 4292 cpu_manager.go:414] "containerMap: removing container" podUID="1f35e32e-1f09-413d-a364-14b8ac1019f2" containerName="main" Oct 30 20:36:43.047145 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:43.047054 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f35e32e-1f09-413d-a364-14b8ac1019f2" containerName="main" Oct 30 20:36:43.047145 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:43.047066 4292 cpu_manager.go:414] "containerMap: removing container" podUID="f562f81a-ebd3-4efa-889d-bf4a757527fe" containerName="main" Oct 30 20:36:43.047145 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:43.047074 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f562f81a-ebd3-4efa-889d-bf4a757527fe" containerName="main" Oct 30 20:36:43.047629 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:43.047180 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="3165f4d2-0fa3-4358-8204-d1e5b46e5d41" containerName="revoke-keys" Oct 30 20:36:43.047629 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:43.047195 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="16def41a-a612-4a4c-ada9-bb170b066af2" containerName="main" Oct 30 20:36:43.047629 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:43.047204 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f35e32e-1f09-413d-a364-14b8ac1019f2" containerName="main" Oct 30 20:36:43.047629 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:43.047216 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="f562f81a-ebd3-4efa-889d-bf4a757527fe" containerName="main" Oct 30 20:36:43.047629 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:43.047230 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0b75746-c06c-407d-8f61-c795c519aad6" containerName="main" Oct 30 20:36:43.051790 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:43.051770 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a65f438e-ctj4f" Oct 30 20:36:43.102651 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:43.102621 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vrb7\" (UniqueName: \"kubernetes.io/projected/17ecf369-9fcc-4bce-a605-2bde479d0bc6-kube-api-access-4vrb7\") pod \"maas-api-revoke-sub-a65f438e-ctj4f\" (UID: \"17ecf369-9fcc-4bce-a605-2bde479d0bc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a65f438e-ctj4f" Oct 30 20:36:43.102651 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:43.102656 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17ecf369-9fcc-4bce-a605-2bde479d0bc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a65f438e-ctj4f\" (UID: \"17ecf369-9fcc-4bce-a605-2bde479d0bc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a65f438e-ctj4f" Oct 30 20:36:43.102894 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:43.102846 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17ecf369-9fcc-4bce-a605-2bde479d0bc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a65f438e-ctj4f\" (UID: \"17ecf369-9fcc-4bce-a605-2bde479d0bc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a65f438e-ctj4f" Oct 30 20:36:43.104664 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:43.104640 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a65f438e-ctj4f"] Oct 30 20:36:43.153216 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:43.153190 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:36:43.153405 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:43.153231 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\"" Oct 30 20:36:43.203837 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:43.203807 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17ecf369-9fcc-4bce-a605-2bde479d0bc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a65f438e-ctj4f\" (UID: \"17ecf369-9fcc-4bce-a605-2bde479d0bc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a65f438e-ctj4f" Oct 30 20:36:43.204001 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:43.203863 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vrb7\" (UniqueName: \"kubernetes.io/projected/17ecf369-9fcc-4bce-a605-2bde479d0bc6-kube-api-access-4vrb7\") pod \"maas-api-revoke-sub-a65f438e-ctj4f\" (UID: \"17ecf369-9fcc-4bce-a605-2bde479d0bc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a65f438e-ctj4f" Oct 30 20:36:43.204001 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:43.203895 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17ecf369-9fcc-4bce-a605-2bde479d0bc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a65f438e-ctj4f\" (UID: \"17ecf369-9fcc-4bce-a605-2bde479d0bc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a65f438e-ctj4f" Oct 30 20:36:43.204587 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:43.204563 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17ecf369-9fcc-4bce-a605-2bde479d0bc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a65f438e-ctj4f\" (UID: \"17ecf369-9fcc-4bce-a605-2bde479d0bc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a65f438e-ctj4f" Oct 30 20:36:43.205994 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:43.205970 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17ecf369-9fcc-4bce-a605-2bde479d0bc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a65f438e-ctj4f\" (UID: \"17ecf369-9fcc-4bce-a605-2bde479d0bc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a65f438e-ctj4f" Oct 30 20:36:43.222572 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:36:43.222528 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:36:43.266216 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:43.266187 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vrb7\" (UniqueName: \"kubernetes.io/projected/17ecf369-9fcc-4bce-a605-2bde479d0bc6-kube-api-access-4vrb7\") pod \"maas-api-revoke-sub-a65f438e-ctj4f\" (UID: \"17ecf369-9fcc-4bce-a605-2bde479d0bc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a65f438e-ctj4f" Oct 30 20:36:43.364475 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:43.364405 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a65f438e-ctj4f" Oct 30 20:36:43.744717 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:43.744683 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a65f438e-ctj4f"] Oct 30 20:36:43.745205 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:36:43.745177 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17ecf369_9fcc_4bce_a605_2bde479d0bc6.slice/crio-62483df232c9344c6e410190573db15a9ae31b41dbdcb32a3f42b62fa6865ec8 WatchSource:0}: Error finding container 62483df232c9344c6e410190573db15a9ae31b41dbdcb32a3f42b62fa6865ec8: Status 404 returned error can't find the container with id 62483df232c9344c6e410190573db15a9ae31b41dbdcb32a3f42b62fa6865ec8 Oct 30 20:36:44.393179 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:44.392306 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-bbfa07-5d784fb568-6ddcf" Oct 30 20:36:44.415547 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:44.415414 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f356d7e-7b1a-4b51-a36f-a809a4a4f91f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f356d7e-7b1a-4b51-a36f-a809a4a4f91f-plugins-config-volume\") pod \"1f356d7e-7b1a-4b51-a36f-a809a4a4f91f\" (UID: \"1f356d7e-7b1a-4b51-a36f-a809a4a4f91f\") " Oct 30 20:36:44.415547 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:44.415478 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f356d7e-7b1a-4b51-a36f-a809a4a4f91f-kube-api-access-lq7ds\" (UniqueName: \"kubernetes.io/projected/1f356d7e-7b1a-4b51-a36f-a809a4a4f91f-kube-api-access-lq7ds\") pod \"1f356d7e-7b1a-4b51-a36f-a809a4a4f91f\" (UID: \"1f356d7e-7b1a-4b51-a36f-a809a4a4f91f\") " Oct 30 20:36:44.415767 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:44.415745 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f356d7e-7b1a-4b51-a36f-a809a4a4f91f-plugins-config-volume" pod "1f356d7e-7b1a-4b51-a36f-a809a4a4f91f" (UID: "1f356d7e-7b1a-4b51-a36f-a809a4a4f91f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:36:44.415933 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:44.415910 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f356d7e-7b1a-4b51-a36f-a809a4a4f91f-plugins-config-volume\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:36:44.417925 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:44.417900 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f356d7e-7b1a-4b51-a36f-a809a4a4f91f-kube-api-access-lq7ds" pod "1f356d7e-7b1a-4b51-a36f-a809a4a4f91f" (UID: "1f356d7e-7b1a-4b51-a36f-a809a4a4f91f"). InnerVolumeSpecName "kube-api-access-lq7ds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:36:44.482133 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:44.482110 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bbfa07-658b9f6ff5-qxnqd" Oct 30 20:36:44.516882 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:44.516853 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0946fd9-55cf-4c53-90af-22020e7b41ac-kube-api-access-qhndq\" (UniqueName: \"kubernetes.io/projected/d0946fd9-55cf-4c53-90af-22020e7b41ac-kube-api-access-qhndq\") pod \"d0946fd9-55cf-4c53-90af-22020e7b41ac\" (UID: \"d0946fd9-55cf-4c53-90af-22020e7b41ac\") " Oct 30 20:36:44.517065 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:44.516957 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0946fd9-55cf-4c53-90af-22020e7b41ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0946fd9-55cf-4c53-90af-22020e7b41ac-plugins-config-volume\") pod \"d0946fd9-55cf-4c53-90af-22020e7b41ac\" (UID: \"d0946fd9-55cf-4c53-90af-22020e7b41ac\") " Oct 30 20:36:44.517263 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:44.517240 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lq7ds\" (UniqueName: \"kubernetes.io/projected/1f356d7e-7b1a-4b51-a36f-a809a4a4f91f-kube-api-access-lq7ds\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:36:44.517386 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:44.517369 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0946fd9-55cf-4c53-90af-22020e7b41ac-plugins-config-volume" pod "d0946fd9-55cf-4c53-90af-22020e7b41ac" (UID: "d0946fd9-55cf-4c53-90af-22020e7b41ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:36:44.549869 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:44.549830 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0946fd9-55cf-4c53-90af-22020e7b41ac-kube-api-access-qhndq" pod "d0946fd9-55cf-4c53-90af-22020e7b41ac" (UID: "d0946fd9-55cf-4c53-90af-22020e7b41ac"). InnerVolumeSpecName "kube-api-access-qhndq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:36:44.597056 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:36:44.597024 4292 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17ecf369_9fcc_4bce_a605_2bde479d0bc6.slice/crio-conmon-17d219b2d958ad656c253317956378eb40dea1e37e9aaa356780455f4586c596.scope\": RecentStats: unable to find data in memory cache]" Oct 30 20:36:44.618270 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:44.618211 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhndq\" (UniqueName: \"kubernetes.io/projected/d0946fd9-55cf-4c53-90af-22020e7b41ac-kube-api-access-qhndq\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:36:44.618270 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:44.618235 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0946fd9-55cf-4c53-90af-22020e7b41ac-plugins-config-volume\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:36:44.744899 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:44.744867 4292 generic.go:356] "Generic (PLEG): container finished" podID="17ecf369-9fcc-4bce-a605-2bde479d0bc6" containerID="17d219b2d958ad656c253317956378eb40dea1e37e9aaa356780455f4586c596" exitCode=0 Oct 30 20:36:44.745404 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:44.744946 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a65f438e-ctj4f" event={"ID":"17ecf369-9fcc-4bce-a605-2bde479d0bc6","Type":"ContainerDied","Data":"17d219b2d958ad656c253317956378eb40dea1e37e9aaa356780455f4586c596"} Oct 30 20:36:44.745404 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:44.744998 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a65f438e-ctj4f" event={"ID":"17ecf369-9fcc-4bce-a605-2bde479d0bc6","Type":"ContainerStarted","Data":"62483df232c9344c6e410190573db15a9ae31b41dbdcb32a3f42b62fa6865ec8"} Oct 30 20:36:44.746346 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:44.746325 4292 generic.go:356] "Generic (PLEG): container finished" podID="1f356d7e-7b1a-4b51-a36f-a809a4a4f91f" containerID="2cffd75e0dbf07dd8029601824695fe3f485818d6b7e0f463051d032b36930e7" exitCode=0 Oct 30 20:36:44.746451 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:44.746382 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-bbfa07-5d784fb568-6ddcf" event={"ID":"1f356d7e-7b1a-4b51-a36f-a809a4a4f91f","Type":"ContainerDied","Data":"2cffd75e0dbf07dd8029601824695fe3f485818d6b7e0f463051d032b36930e7"} Oct 30 20:36:44.746451 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:44.746422 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-bbfa07-5d784fb568-6ddcf" event={"ID":"1f356d7e-7b1a-4b51-a36f-a809a4a4f91f","Type":"ContainerDied","Data":"e5cba0945f72777adc360d587d966c130dbd755103e38a853715a2052379339b"} Oct 30 20:36:44.746451 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:44.746387 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-bbfa07-5d784fb568-6ddcf" Oct 30 20:36:44.746451 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:44.746444 4292 scope.go:122] "RemoveContainer" containerID="2cffd75e0dbf07dd8029601824695fe3f485818d6b7e0f463051d032b36930e7" Oct 30 20:36:44.747778 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:44.747761 4292 generic.go:356] "Generic (PLEG): container finished" podID="d0946fd9-55cf-4c53-90af-22020e7b41ac" containerID="ca77cb60489d7c4ab5a0997b1863d00a80ca659ce6d7f37b9a877f6295747064" exitCode=0 Oct 30 20:36:44.747892 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:44.747791 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bbfa07-658b9f6ff5-qxnqd" event={"ID":"d0946fd9-55cf-4c53-90af-22020e7b41ac","Type":"ContainerDied","Data":"ca77cb60489d7c4ab5a0997b1863d00a80ca659ce6d7f37b9a877f6295747064"} Oct 30 20:36:44.747892 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:44.747815 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bbfa07-658b9f6ff5-qxnqd" Oct 30 20:36:44.747996 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:44.747816 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bbfa07-658b9f6ff5-qxnqd" event={"ID":"d0946fd9-55cf-4c53-90af-22020e7b41ac","Type":"ContainerDied","Data":"d1edd8624ed35416998c8c8a2caa045d1e6b3936b9859befe11cfce3b687678b"} Oct 30 20:36:44.757060 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:44.757044 4292 scope.go:122] "RemoveContainer" containerID="2cffd75e0dbf07dd8029601824695fe3f485818d6b7e0f463051d032b36930e7" Oct 30 20:36:44.757329 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:36:44.757311 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cffd75e0dbf07dd8029601824695fe3f485818d6b7e0f463051d032b36930e7\": container with ID starting with 2cffd75e0dbf07dd8029601824695fe3f485818d6b7e0f463051d032b36930e7 not found: ID does not exist" containerID="2cffd75e0dbf07dd8029601824695fe3f485818d6b7e0f463051d032b36930e7" Oct 30 20:36:44.757469 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:44.757370 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cffd75e0dbf07dd8029601824695fe3f485818d6b7e0f463051d032b36930e7"} err="failed to get container status \"2cffd75e0dbf07dd8029601824695fe3f485818d6b7e0f463051d032b36930e7\": rpc error: code = NotFound desc = could not find container \"2cffd75e0dbf07dd8029601824695fe3f485818d6b7e0f463051d032b36930e7\": container with ID starting with 2cffd75e0dbf07dd8029601824695fe3f485818d6b7e0f463051d032b36930e7 not found: ID does not exist" Oct 30 20:36:44.757469 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:44.757396 4292 scope.go:122] "RemoveContainer" containerID="ca77cb60489d7c4ab5a0997b1863d00a80ca659ce6d7f37b9a877f6295747064" Oct 30 20:36:44.765434 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:44.765419 4292 scope.go:122] "RemoveContainer" containerID="ca77cb60489d7c4ab5a0997b1863d00a80ca659ce6d7f37b9a877f6295747064" Oct 30 20:36:44.765686 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:36:44.765666 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca77cb60489d7c4ab5a0997b1863d00a80ca659ce6d7f37b9a877f6295747064\": container with ID starting with ca77cb60489d7c4ab5a0997b1863d00a80ca659ce6d7f37b9a877f6295747064 not found: ID does not exist" containerID="ca77cb60489d7c4ab5a0997b1863d00a80ca659ce6d7f37b9a877f6295747064" Oct 30 20:36:44.765782 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:44.765690 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca77cb60489d7c4ab5a0997b1863d00a80ca659ce6d7f37b9a877f6295747064"} err="failed to get container status \"ca77cb60489d7c4ab5a0997b1863d00a80ca659ce6d7f37b9a877f6295747064\": rpc error: code = NotFound desc = could not find container \"ca77cb60489d7c4ab5a0997b1863d00a80ca659ce6d7f37b9a877f6295747064\": container with ID starting with ca77cb60489d7c4ab5a0997b1863d00a80ca659ce6d7f37b9a877f6295747064 not found: ID does not exist" Oct 30 20:36:44.994123 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:44.984625 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-bbfa07-5d784fb568-6ddcf"] Oct 30 20:36:44.994123 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:44.987556 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-bbfa07-5d784fb568-6ddcf"] Oct 30 20:36:45.096662 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:45.096626 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-bbfa07-658b9f6ff5-qxnqd"] Oct 30 20:36:45.100858 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:45.100833 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-bbfa07-658b9f6ff5-qxnqd"] Oct 30 20:36:45.892281 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:45.892259 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a65f438e-ctj4f" Oct 30 20:36:45.931475 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:45.931451 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17ecf369-9fcc-4bce-a605-2bde479d0bc6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17ecf369-9fcc-4bce-a605-2bde479d0bc6-maas-api-xks-ca\") pod \"17ecf369-9fcc-4bce-a605-2bde479d0bc6\" (UID: \"17ecf369-9fcc-4bce-a605-2bde479d0bc6\") " Oct 30 20:36:45.931617 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:45.931506 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17ecf369-9fcc-4bce-a605-2bde479d0bc6-kube-api-access-4vrb7\" (UniqueName: \"kubernetes.io/projected/17ecf369-9fcc-4bce-a605-2bde479d0bc6-kube-api-access-4vrb7\") pod \"17ecf369-9fcc-4bce-a605-2bde479d0bc6\" (UID: \"17ecf369-9fcc-4bce-a605-2bde479d0bc6\") " Oct 30 20:36:45.931617 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:45.931536 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17ecf369-9fcc-4bce-a605-2bde479d0bc6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17ecf369-9fcc-4bce-a605-2bde479d0bc6-maas-api-openshift-ca\") pod \"17ecf369-9fcc-4bce-a605-2bde479d0bc6\" (UID: \"17ecf369-9fcc-4bce-a605-2bde479d0bc6\") " Oct 30 20:36:45.931934 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:45.931912 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17ecf369-9fcc-4bce-a605-2bde479d0bc6-maas-api-openshift-ca" pod "17ecf369-9fcc-4bce-a605-2bde479d0bc6" (UID: "17ecf369-9fcc-4bce-a605-2bde479d0bc6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:36:45.933585 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:45.933565 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17ecf369-9fcc-4bce-a605-2bde479d0bc6-maas-api-xks-ca" pod "17ecf369-9fcc-4bce-a605-2bde479d0bc6" (UID: "17ecf369-9fcc-4bce-a605-2bde479d0bc6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:36:45.933639 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:45.933584 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17ecf369-9fcc-4bce-a605-2bde479d0bc6-kube-api-access-4vrb7" pod "17ecf369-9fcc-4bce-a605-2bde479d0bc6" (UID: "17ecf369-9fcc-4bce-a605-2bde479d0bc6"). InnerVolumeSpecName "kube-api-access-4vrb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:36:46.032951 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:46.032925 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17ecf369-9fcc-4bce-a605-2bde479d0bc6-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:36:46.033089 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:46.032949 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vrb7\" (UniqueName: \"kubernetes.io/projected/17ecf369-9fcc-4bce-a605-2bde479d0bc6-kube-api-access-4vrb7\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:36:46.033089 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:46.032975 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17ecf369-9fcc-4bce-a605-2bde479d0bc6-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:36:46.581427 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:46.581393 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f356d7e-7b1a-4b51-a36f-a809a4a4f91f" path="/var/lib/kubelet/pods/1f356d7e-7b1a-4b51-a36f-a809a4a4f91f/volumes" Oct 30 20:36:46.581796 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:46.581783 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0946fd9-55cf-4c53-90af-22020e7b41ac" path="/var/lib/kubelet/pods/d0946fd9-55cf-4c53-90af-22020e7b41ac/volumes" Oct 30 20:36:46.760330 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:46.760300 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a65f438e-ctj4f" Oct 30 20:36:46.760535 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:46.760299 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a65f438e-ctj4f" event={"ID":"17ecf369-9fcc-4bce-a605-2bde479d0bc6","Type":"ContainerDied","Data":"62483df232c9344c6e410190573db15a9ae31b41dbdcb32a3f42b62fa6865ec8"} Oct 30 20:36:46.760535 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:46.760430 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62483df232c9344c6e410190573db15a9ae31b41dbdcb32a3f42b62fa6865ec8" Oct 30 20:36:48.732015 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:48.731974 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e391de-7b69fb5d95-gtqdh"] Oct 30 20:36:48.771692 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:48.771650 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e391de-7b69fb5d95-gtqdh" podUID="9d0ec8d7-e0d4-4452-b263-89a3239af156" containerName="maas-api" containerID="cri-o://8670f3d3277c9ae30c884994daae95e4a0f3ad13640180ee2cc1a74d183d97f8" gracePeriod=30 Oct 30 20:36:49.071143 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:49.071114 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e391de-7b69fb5d95-gtqdh" Oct 30 20:36:49.160950 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:49.160920 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d0ec8d7-e0d4-4452-b263-89a3239af156-kube-api-access-22kdq\" (UniqueName: \"kubernetes.io/projected/9d0ec8d7-e0d4-4452-b263-89a3239af156-kube-api-access-22kdq\") pod \"9d0ec8d7-e0d4-4452-b263-89a3239af156\" (UID: \"9d0ec8d7-e0d4-4452-b263-89a3239af156\") " Oct 30 20:36:49.161147 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:49.161010 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d0ec8d7-e0d4-4452-b263-89a3239af156-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d0ec8d7-e0d4-4452-b263-89a3239af156-openshift-service-ca\") pod \"9d0ec8d7-e0d4-4452-b263-89a3239af156\" (UID: \"9d0ec8d7-e0d4-4452-b263-89a3239af156\") " Oct 30 20:36:49.161147 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:49.161033 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d0ec8d7-e0d4-4452-b263-89a3239af156-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d0ec8d7-e0d4-4452-b263-89a3239af156-maas-api-tls\") pod \"9d0ec8d7-e0d4-4452-b263-89a3239af156\" (UID: \"9d0ec8d7-e0d4-4452-b263-89a3239af156\") " Oct 30 20:36:49.161147 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:49.161062 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d0ec8d7-e0d4-4452-b263-89a3239af156-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d0ec8d7-e0d4-4452-b263-89a3239af156-metrics-certs\") pod \"9d0ec8d7-e0d4-4452-b263-89a3239af156\" (UID: \"9d0ec8d7-e0d4-4452-b263-89a3239af156\") " Oct 30 20:36:49.161390 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:49.161331 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d0ec8d7-e0d4-4452-b263-89a3239af156-openshift-service-ca" pod "9d0ec8d7-e0d4-4452-b263-89a3239af156" (UID: "9d0ec8d7-e0d4-4452-b263-89a3239af156"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:36:49.163204 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:49.163179 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d0ec8d7-e0d4-4452-b263-89a3239af156-kube-api-access-22kdq" pod "9d0ec8d7-e0d4-4452-b263-89a3239af156" (UID: "9d0ec8d7-e0d4-4452-b263-89a3239af156"). InnerVolumeSpecName "kube-api-access-22kdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:36:49.163204 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:49.163195 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d0ec8d7-e0d4-4452-b263-89a3239af156-metrics-certs" pod "9d0ec8d7-e0d4-4452-b263-89a3239af156" (UID: "9d0ec8d7-e0d4-4452-b263-89a3239af156"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:36:49.163388 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:49.163291 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d0ec8d7-e0d4-4452-b263-89a3239af156-maas-api-tls" pod "9d0ec8d7-e0d4-4452-b263-89a3239af156" (UID: "9d0ec8d7-e0d4-4452-b263-89a3239af156"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:36:49.262091 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:49.262059 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22kdq\" (UniqueName: \"kubernetes.io/projected/9d0ec8d7-e0d4-4452-b263-89a3239af156-kube-api-access-22kdq\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:36:49.262091 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:49.262089 4292 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d0ec8d7-e0d4-4452-b263-89a3239af156-openshift-service-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:36:49.262305 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:49.262102 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d0ec8d7-e0d4-4452-b263-89a3239af156-maas-api-tls\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:36:49.262305 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:49.262112 4292 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d0ec8d7-e0d4-4452-b263-89a3239af156-metrics-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:36:49.779453 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:49.779418 4292 generic.go:356] "Generic (PLEG): container finished" podID="9d0ec8d7-e0d4-4452-b263-89a3239af156" containerID="8670f3d3277c9ae30c884994daae95e4a0f3ad13640180ee2cc1a74d183d97f8" exitCode=0 Oct 30 20:36:49.779820 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:49.779464 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e391de-7b69fb5d95-gtqdh" Oct 30 20:36:49.779820 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:49.779492 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e391de-7b69fb5d95-gtqdh" event={"ID":"9d0ec8d7-e0d4-4452-b263-89a3239af156","Type":"ContainerDied","Data":"8670f3d3277c9ae30c884994daae95e4a0f3ad13640180ee2cc1a74d183d97f8"} Oct 30 20:36:49.779820 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:49.779545 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e391de-7b69fb5d95-gtqdh" event={"ID":"9d0ec8d7-e0d4-4452-b263-89a3239af156","Type":"ContainerDied","Data":"ccc8af43c8c44cc14956d6f622ffe1ba713383e804fe9583e806c93fadba6966"} Oct 30 20:36:49.779820 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:49.779566 4292 scope.go:122] "RemoveContainer" containerID="8670f3d3277c9ae30c884994daae95e4a0f3ad13640180ee2cc1a74d183d97f8" Oct 30 20:36:49.789448 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:49.789429 4292 scope.go:122] "RemoveContainer" containerID="8670f3d3277c9ae30c884994daae95e4a0f3ad13640180ee2cc1a74d183d97f8" Oct 30 20:36:49.789727 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:36:49.789704 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8670f3d3277c9ae30c884994daae95e4a0f3ad13640180ee2cc1a74d183d97f8\": container with ID starting with 8670f3d3277c9ae30c884994daae95e4a0f3ad13640180ee2cc1a74d183d97f8 not found: ID does not exist" containerID="8670f3d3277c9ae30c884994daae95e4a0f3ad13640180ee2cc1a74d183d97f8" Oct 30 20:36:49.789825 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:49.789732 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8670f3d3277c9ae30c884994daae95e4a0f3ad13640180ee2cc1a74d183d97f8"} err="failed to get container status \"8670f3d3277c9ae30c884994daae95e4a0f3ad13640180ee2cc1a74d183d97f8\": rpc error: code = NotFound desc = could not find container \"8670f3d3277c9ae30c884994daae95e4a0f3ad13640180ee2cc1a74d183d97f8\": container with ID starting with 8670f3d3277c9ae30c884994daae95e4a0f3ad13640180ee2cc1a74d183d97f8 not found: ID does not exist" Oct 30 20:36:49.946657 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:49.946625 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e391de-7b69fb5d95-gtqdh"] Oct 30 20:36:49.999919 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:49.999882 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e391de-7b69fb5d95-gtqdh"] Oct 30 20:36:50.082686 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:50.082612 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b6ecbe3-hw7jt"] Oct 30 20:36:50.083003 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:50.082991 4292 cpu_manager.go:414] "containerMap: removing container" podUID="9d0ec8d7-e0d4-4452-b263-89a3239af156" containerName="maas-api" Oct 30 20:36:50.083044 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:50.083004 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d0ec8d7-e0d4-4452-b263-89a3239af156" containerName="maas-api" Oct 30 20:36:50.083044 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:50.083017 4292 cpu_manager.go:414] "containerMap: removing container" podUID="1f356d7e-7b1a-4b51-a36f-a809a4a4f91f" containerName="payload-processing" Oct 30 20:36:50.083044 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:50.083023 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f356d7e-7b1a-4b51-a36f-a809a4a4f91f" containerName="payload-processing" Oct 30 20:36:50.083044 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:50.083034 4292 cpu_manager.go:414] "containerMap: removing container" podUID="d0946fd9-55cf-4c53-90af-22020e7b41ac" containerName="payload-pre-processing" Oct 30 20:36:50.083044 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:50.083039 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0946fd9-55cf-4c53-90af-22020e7b41ac" containerName="payload-pre-processing" Oct 30 20:36:50.083194 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:50.083056 4292 cpu_manager.go:414] "containerMap: removing container" podUID="17ecf369-9fcc-4bce-a605-2bde479d0bc6" containerName="revoke-keys" Oct 30 20:36:50.083194 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:50.083062 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17ecf369-9fcc-4bce-a605-2bde479d0bc6" containerName="revoke-keys" Oct 30 20:36:50.083194 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:50.083109 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0946fd9-55cf-4c53-90af-22020e7b41ac" containerName="payload-pre-processing" Oct 30 20:36:50.083194 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:50.083120 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="17ecf369-9fcc-4bce-a605-2bde479d0bc6" containerName="revoke-keys" Oct 30 20:36:50.083194 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:50.083127 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d0ec8d7-e0d4-4452-b263-89a3239af156" containerName="maas-api" Oct 30 20:36:50.083194 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:50.083134 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f356d7e-7b1a-4b51-a36f-a809a4a4f91f" containerName="payload-processing" Oct 30 20:36:50.087395 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:50.087379 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6ecbe3-hw7jt" Oct 30 20:36:50.152671 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:50.152631 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce7c24b1-z7zcg"] Oct 30 20:36:50.155569 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:50.155545 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:36:50.155720 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:50.155663 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\"" Oct 30 20:36:50.156697 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:50.156678 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b6ecbe3-hw7jt"] Oct 30 20:36:50.156773 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:50.156764 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce7c24b1-z7zcg" Oct 30 20:36:50.169171 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:50.169143 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg5p2\" (UniqueName: \"kubernetes.io/projected/981830dd-2c55-4876-be0a-84eb1dfb69ae-kube-api-access-sg5p2\") pod \"maas-api-revoke-sub-3b6ecbe3-hw7jt\" (UID: \"981830dd-2c55-4876-be0a-84eb1dfb69ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6ecbe3-hw7jt" Oct 30 20:36:50.169171 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:50.169175 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/981830dd-2c55-4876-be0a-84eb1dfb69ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b6ecbe3-hw7jt\" (UID: \"981830dd-2c55-4876-be0a-84eb1dfb69ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6ecbe3-hw7jt" Oct 30 20:36:50.169412 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:50.169315 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/981830dd-2c55-4876-be0a-84eb1dfb69ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b6ecbe3-hw7jt\" (UID: \"981830dd-2c55-4876-be0a-84eb1dfb69ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6ecbe3-hw7jt" Oct 30 20:36:50.211125 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:50.211097 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce7c24b1-z7zcg"] Oct 30 20:36:50.270379 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:50.270329 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/981830dd-2c55-4876-be0a-84eb1dfb69ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b6ecbe3-hw7jt\" (UID: \"981830dd-2c55-4876-be0a-84eb1dfb69ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6ecbe3-hw7jt" Oct 30 20:36:50.270581 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:50.270406 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7af30adc-8334-4ea8-bc21-e61f98011790-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce7c24b1-z7zcg\" (UID: \"7af30adc-8334-4ea8-bc21-e61f98011790\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce7c24b1-z7zcg" Oct 30 20:36:50.270581 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:50.270507 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg5p2\" (UniqueName: \"kubernetes.io/projected/981830dd-2c55-4876-be0a-84eb1dfb69ae-kube-api-access-sg5p2\") pod \"maas-api-revoke-sub-3b6ecbe3-hw7jt\" (UID: \"981830dd-2c55-4876-be0a-84eb1dfb69ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6ecbe3-hw7jt" Oct 30 20:36:50.270581 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:50.270548 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/981830dd-2c55-4876-be0a-84eb1dfb69ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b6ecbe3-hw7jt\" (UID: \"981830dd-2c55-4876-be0a-84eb1dfb69ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6ecbe3-hw7jt" Oct 30 20:36:50.270761 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:50.270611 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvnl8\" (UniqueName: \"kubernetes.io/projected/7af30adc-8334-4ea8-bc21-e61f98011790-kube-api-access-xvnl8\") pod \"maas-api-revoke-sub-ce7c24b1-z7zcg\" (UID: \"7af30adc-8334-4ea8-bc21-e61f98011790\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce7c24b1-z7zcg" Oct 30 20:36:50.270761 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:50.270695 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7af30adc-8334-4ea8-bc21-e61f98011790-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce7c24b1-z7zcg\" (UID: \"7af30adc-8334-4ea8-bc21-e61f98011790\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce7c24b1-z7zcg" Oct 30 20:36:50.271691 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:50.271665 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/981830dd-2c55-4876-be0a-84eb1dfb69ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b6ecbe3-hw7jt\" (UID: \"981830dd-2c55-4876-be0a-84eb1dfb69ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6ecbe3-hw7jt" Oct 30 20:36:50.272777 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:50.272758 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/981830dd-2c55-4876-be0a-84eb1dfb69ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b6ecbe3-hw7jt\" (UID: \"981830dd-2c55-4876-be0a-84eb1dfb69ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6ecbe3-hw7jt" Oct 30 20:36:50.335646 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:50.335581 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg5p2\" (UniqueName: \"kubernetes.io/projected/981830dd-2c55-4876-be0a-84eb1dfb69ae-kube-api-access-sg5p2\") pod \"maas-api-revoke-sub-3b6ecbe3-hw7jt\" (UID: \"981830dd-2c55-4876-be0a-84eb1dfb69ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6ecbe3-hw7jt" Oct 30 20:36:50.371799 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:50.371768 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7af30adc-8334-4ea8-bc21-e61f98011790-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce7c24b1-z7zcg\" (UID: \"7af30adc-8334-4ea8-bc21-e61f98011790\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce7c24b1-z7zcg" Oct 30 20:36:50.371932 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:50.371823 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7af30adc-8334-4ea8-bc21-e61f98011790-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce7c24b1-z7zcg\" (UID: \"7af30adc-8334-4ea8-bc21-e61f98011790\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce7c24b1-z7zcg" Oct 30 20:36:50.371932 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:50.371876 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvnl8\" (UniqueName: \"kubernetes.io/projected/7af30adc-8334-4ea8-bc21-e61f98011790-kube-api-access-xvnl8\") pod \"maas-api-revoke-sub-ce7c24b1-z7zcg\" (UID: \"7af30adc-8334-4ea8-bc21-e61f98011790\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce7c24b1-z7zcg" Oct 30 20:36:50.372396 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:50.372376 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7af30adc-8334-4ea8-bc21-e61f98011790-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce7c24b1-z7zcg\" (UID: \"7af30adc-8334-4ea8-bc21-e61f98011790\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce7c24b1-z7zcg" Oct 30 20:36:50.374059 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:50.374027 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7af30adc-8334-4ea8-bc21-e61f98011790-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce7c24b1-z7zcg\" (UID: \"7af30adc-8334-4ea8-bc21-e61f98011790\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce7c24b1-z7zcg" Oct 30 20:36:50.400026 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:50.400000 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6ecbe3-hw7jt" Oct 30 20:36:50.436220 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:50.436192 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvnl8\" (UniqueName: \"kubernetes.io/projected/7af30adc-8334-4ea8-bc21-e61f98011790-kube-api-access-xvnl8\") pod \"maas-api-revoke-sub-ce7c24b1-z7zcg\" (UID: \"7af30adc-8334-4ea8-bc21-e61f98011790\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce7c24b1-z7zcg" Oct 30 20:36:50.469697 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:50.469666 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce7c24b1-z7zcg" Oct 30 20:36:50.582278 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:50.582246 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d0ec8d7-e0d4-4452-b263-89a3239af156" path="/var/lib/kubelet/pods/9d0ec8d7-e0d4-4452-b263-89a3239af156/volumes" Oct 30 20:36:50.779271 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:50.779248 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b6ecbe3-hw7jt"] Oct 30 20:36:50.780879 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:36:50.780851 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod981830dd_2c55_4876_be0a_84eb1dfb69ae.slice/crio-d02ba1e55fb50eff26526b7d9b4f124a187aebb964b8fbff9b9d27271757eea8 WatchSource:0}: Error finding container d02ba1e55fb50eff26526b7d9b4f124a187aebb964b8fbff9b9d27271757eea8: Status 404 returned error can't find the container with id d02ba1e55fb50eff26526b7d9b4f124a187aebb964b8fbff9b9d27271757eea8 Oct 30 20:36:50.882786 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:50.882762 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce7c24b1-z7zcg"] Oct 30 20:36:50.884972 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:36:50.884944 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7af30adc_8334_4ea8_bc21_e61f98011790.slice/crio-992a8ae1de629d685c0ec689653d8b1540af867824813bcead48ab9a96601d6a WatchSource:0}: Error finding container 992a8ae1de629d685c0ec689653d8b1540af867824813bcead48ab9a96601d6a: Status 404 returned error can't find the container with id 992a8ae1de629d685c0ec689653d8b1540af867824813bcead48ab9a96601d6a Oct 30 20:36:50.999297 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:50.999271 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e391de-z9ghb"] Oct 30 20:36:51.001919 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:51.001899 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e391de-z9ghb"] Oct 30 20:36:51.144373 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:51.144312 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-98a19b-779f877d8-hgncx" podUID="e8810ca2-97ba-4a61-84da-fc464be80c17" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:36:51.793154 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:51.793122 4292 generic.go:356] "Generic (PLEG): container finished" podID="7af30adc-8334-4ea8-bc21-e61f98011790" containerID="0002a3d2eead2a497cdf9eec96246ae98340c7e38e7fb2b6c0d14b594cec8dce" exitCode=0 Oct 30 20:36:51.793582 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:51.793210 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce7c24b1-z7zcg" event={"ID":"7af30adc-8334-4ea8-bc21-e61f98011790","Type":"ContainerDied","Data":"0002a3d2eead2a497cdf9eec96246ae98340c7e38e7fb2b6c0d14b594cec8dce"} Oct 30 20:36:51.793582 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:51.793267 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce7c24b1-z7zcg" event={"ID":"7af30adc-8334-4ea8-bc21-e61f98011790","Type":"ContainerStarted","Data":"992a8ae1de629d685c0ec689653d8b1540af867824813bcead48ab9a96601d6a"} Oct 30 20:36:51.794864 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:51.794841 4292 generic.go:356] "Generic (PLEG): container finished" podID="981830dd-2c55-4876-be0a-84eb1dfb69ae" containerID="72203eb220235d5802b6bb22ec9ded1ff88c64dc00c9d3f8534399e365faf983" exitCode=0 Oct 30 20:36:51.794976 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:51.794920 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6ecbe3-hw7jt" event={"ID":"981830dd-2c55-4876-be0a-84eb1dfb69ae","Type":"ContainerDied","Data":"72203eb220235d5802b6bb22ec9ded1ff88c64dc00c9d3f8534399e365faf983"} Oct 30 20:36:51.794976 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:51.794962 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6ecbe3-hw7jt" event={"ID":"981830dd-2c55-4876-be0a-84eb1dfb69ae","Type":"ContainerStarted","Data":"d02ba1e55fb50eff26526b7d9b4f124a187aebb964b8fbff9b9d27271757eea8"} Oct 30 20:36:52.581598 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:52.581568 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="720c0839-9035-4182-a45c-0fc081686e0c" path="/var/lib/kubelet/pods/720c0839-9035-4182-a45c-0fc081686e0c/volumes" Oct 30 20:36:52.937982 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:52.937957 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce7c24b1-z7zcg" Oct 30 20:36:52.967292 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:52.967267 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6ecbe3-hw7jt" Oct 30 20:36:52.997668 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:52.997639 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7af30adc-8334-4ea8-bc21-e61f98011790-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7af30adc-8334-4ea8-bc21-e61f98011790-maas-api-xks-ca\") pod \"7af30adc-8334-4ea8-bc21-e61f98011790\" (UID: \"7af30adc-8334-4ea8-bc21-e61f98011790\") " Oct 30 20:36:52.997793 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:52.997681 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/981830dd-2c55-4876-be0a-84eb1dfb69ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/981830dd-2c55-4876-be0a-84eb1dfb69ae-maas-api-openshift-ca\") pod \"981830dd-2c55-4876-be0a-84eb1dfb69ae\" (UID: \"981830dd-2c55-4876-be0a-84eb1dfb69ae\") " Oct 30 20:36:52.997793 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:52.997733 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/981830dd-2c55-4876-be0a-84eb1dfb69ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/981830dd-2c55-4876-be0a-84eb1dfb69ae-maas-api-xks-ca\") pod \"981830dd-2c55-4876-be0a-84eb1dfb69ae\" (UID: \"981830dd-2c55-4876-be0a-84eb1dfb69ae\") " Oct 30 20:36:52.997793 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:52.997787 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7af30adc-8334-4ea8-bc21-e61f98011790-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7af30adc-8334-4ea8-bc21-e61f98011790-maas-api-openshift-ca\") pod \"7af30adc-8334-4ea8-bc21-e61f98011790\" (UID: \"7af30adc-8334-4ea8-bc21-e61f98011790\") " Oct 30 20:36:52.997931 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:52.997845 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7af30adc-8334-4ea8-bc21-e61f98011790-kube-api-access-xvnl8\" (UniqueName: \"kubernetes.io/projected/7af30adc-8334-4ea8-bc21-e61f98011790-kube-api-access-xvnl8\") pod \"7af30adc-8334-4ea8-bc21-e61f98011790\" (UID: \"7af30adc-8334-4ea8-bc21-e61f98011790\") " Oct 30 20:36:52.997931 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:52.997892 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/981830dd-2c55-4876-be0a-84eb1dfb69ae-kube-api-access-sg5p2\" (UniqueName: \"kubernetes.io/projected/981830dd-2c55-4876-be0a-84eb1dfb69ae-kube-api-access-sg5p2\") pod \"981830dd-2c55-4876-be0a-84eb1dfb69ae\" (UID: \"981830dd-2c55-4876-be0a-84eb1dfb69ae\") " Oct 30 20:36:52.998153 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:52.998123 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/981830dd-2c55-4876-be0a-84eb1dfb69ae-maas-api-openshift-ca" pod "981830dd-2c55-4876-be0a-84eb1dfb69ae" (UID: "981830dd-2c55-4876-be0a-84eb1dfb69ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:36:52.998295 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:52.998175 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7af30adc-8334-4ea8-bc21-e61f98011790-maas-api-openshift-ca" pod "7af30adc-8334-4ea8-bc21-e61f98011790" (UID: "7af30adc-8334-4ea8-bc21-e61f98011790"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:36:52.998392 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:52.998305 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7af30adc-8334-4ea8-bc21-e61f98011790-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:36:52.998392 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:52.998321 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/981830dd-2c55-4876-be0a-84eb1dfb69ae-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:36:52.999860 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:52.999838 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/981830dd-2c55-4876-be0a-84eb1dfb69ae-maas-api-xks-ca" pod "981830dd-2c55-4876-be0a-84eb1dfb69ae" (UID: "981830dd-2c55-4876-be0a-84eb1dfb69ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:36:53.000013 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:52.999996 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/981830dd-2c55-4876-be0a-84eb1dfb69ae-kube-api-access-sg5p2" pod "981830dd-2c55-4876-be0a-84eb1dfb69ae" (UID: "981830dd-2c55-4876-be0a-84eb1dfb69ae"). InnerVolumeSpecName "kube-api-access-sg5p2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:36:53.000268 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:53.000238 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7af30adc-8334-4ea8-bc21-e61f98011790-maas-api-xks-ca" pod "7af30adc-8334-4ea8-bc21-e61f98011790" (UID: "7af30adc-8334-4ea8-bc21-e61f98011790"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:36:53.000382 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:53.000248 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7af30adc-8334-4ea8-bc21-e61f98011790-kube-api-access-xvnl8" pod "7af30adc-8334-4ea8-bc21-e61f98011790" (UID: "7af30adc-8334-4ea8-bc21-e61f98011790"). InnerVolumeSpecName "kube-api-access-xvnl8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:36:53.098955 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:53.098921 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7af30adc-8334-4ea8-bc21-e61f98011790-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:36:53.098955 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:53.098946 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/981830dd-2c55-4876-be0a-84eb1dfb69ae-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:36:53.098955 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:53.098956 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvnl8\" (UniqueName: \"kubernetes.io/projected/7af30adc-8334-4ea8-bc21-e61f98011790-kube-api-access-xvnl8\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:36:53.099185 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:53.098966 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg5p2\" (UniqueName: \"kubernetes.io/projected/981830dd-2c55-4876-be0a-84eb1dfb69ae-kube-api-access-sg5p2\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:36:53.807036 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:53.807006 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce7c24b1-z7zcg" Oct 30 20:36:53.807231 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:53.807043 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce7c24b1-z7zcg" event={"ID":"7af30adc-8334-4ea8-bc21-e61f98011790","Type":"ContainerDied","Data":"992a8ae1de629d685c0ec689653d8b1540af867824813bcead48ab9a96601d6a"} Oct 30 20:36:53.807231 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:53.807100 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="992a8ae1de629d685c0ec689653d8b1540af867824813bcead48ab9a96601d6a" Oct 30 20:36:53.808398 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:53.808378 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6ecbe3-hw7jt" Oct 30 20:36:53.808507 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:53.808406 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6ecbe3-hw7jt" event={"ID":"981830dd-2c55-4876-be0a-84eb1dfb69ae","Type":"ContainerDied","Data":"d02ba1e55fb50eff26526b7d9b4f124a187aebb964b8fbff9b9d27271757eea8"} Oct 30 20:36:53.808507 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:53.808450 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d02ba1e55fb50eff26526b7d9b4f124a187aebb964b8fbff9b9d27271757eea8" Oct 30 20:36:58.061907 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:58.061868 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms"] Oct 30 20:36:58.062441 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:58.062073 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms" podUID="9d5226bf-9ab0-496b-b2f5-f0d730554365" containerName="istio-proxy" containerID="cri-o://41e0a2b5abd2268e6d14c9a1757705f4d392117878465c05d42f78d09b9aeb25" gracePeriod=30 Oct 30 20:36:58.339482 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:58.339408 4292 scope.go:122] "RemoveContainer" containerID="4d7d9eaef73b4abb2270be44990a74ec18acdee9692f41ea49b3897d603ebc94" Oct 30 20:36:58.990158 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:58.990123 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-547082-hfgnm"] Oct 30 20:36:58.990530 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:58.990516 4292 cpu_manager.go:414] "containerMap: removing container" podUID="7af30adc-8334-4ea8-bc21-e61f98011790" containerName="revoke-keys" Oct 30 20:36:58.990591 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:58.990532 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7af30adc-8334-4ea8-bc21-e61f98011790" containerName="revoke-keys" Oct 30 20:36:58.990591 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:58.990555 4292 cpu_manager.go:414] "containerMap: removing container" podUID="981830dd-2c55-4876-be0a-84eb1dfb69ae" containerName="revoke-keys" Oct 30 20:36:58.990591 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:58.990576 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="981830dd-2c55-4876-be0a-84eb1dfb69ae" containerName="revoke-keys" Oct 30 20:36:58.990689 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:58.990634 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="981830dd-2c55-4876-be0a-84eb1dfb69ae" containerName="revoke-keys" Oct 30 20:36:58.990689 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:58.990645 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="7af30adc-8334-4ea8-bc21-e61f98011790" containerName="revoke-keys" Oct 30 20:36:58.995301 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:58.995283 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-547082-hfgnm" Oct 30 20:36:59.049321 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:59.049281 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-547082-hfgnm"] Oct 30 20:36:59.056468 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:59.056434 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25776845-32c3-4c70-aa99-26fe1fa5ecd0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-547082-hfgnm\" (UID: \"25776845-32c3-4c70-aa99-26fe1fa5ecd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-547082-hfgnm" Oct 30 20:36:59.056929 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:59.056471 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqnvp\" (UniqueName: \"kubernetes.io/projected/25776845-32c3-4c70-aa99-26fe1fa5ecd0-kube-api-access-lqnvp\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-547082-hfgnm\" (UID: \"25776845-32c3-4c70-aa99-26fe1fa5ecd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-547082-hfgnm" Oct 30 20:36:59.056929 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:59.056644 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25776845-32c3-4c70-aa99-26fe1fa5ecd0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-547082-hfgnm\" (UID: \"25776845-32c3-4c70-aa99-26fe1fa5ecd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-547082-hfgnm" Oct 30 20:36:59.097139 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:59.097104 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:36:59.097530 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:59.097202 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\"" Oct 30 20:36:59.158128 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:59.158094 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25776845-32c3-4c70-aa99-26fe1fa5ecd0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-547082-hfgnm\" (UID: \"25776845-32c3-4c70-aa99-26fe1fa5ecd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-547082-hfgnm" Oct 30 20:36:59.158296 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:59.158140 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25776845-32c3-4c70-aa99-26fe1fa5ecd0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-547082-hfgnm\" (UID: \"25776845-32c3-4c70-aa99-26fe1fa5ecd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-547082-hfgnm" Oct 30 20:36:59.158296 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:59.158164 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqnvp\" (UniqueName: \"kubernetes.io/projected/25776845-32c3-4c70-aa99-26fe1fa5ecd0-kube-api-access-lqnvp\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-547082-hfgnm\" (UID: \"25776845-32c3-4c70-aa99-26fe1fa5ecd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-547082-hfgnm" Oct 30 20:36:59.158971 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:59.158905 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25776845-32c3-4c70-aa99-26fe1fa5ecd0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-547082-hfgnm\" (UID: \"25776845-32c3-4c70-aa99-26fe1fa5ecd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-547082-hfgnm" Oct 30 20:36:59.160605 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:59.160586 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25776845-32c3-4c70-aa99-26fe1fa5ecd0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-547082-hfgnm\" (UID: \"25776845-32c3-4c70-aa99-26fe1fa5ecd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-547082-hfgnm" Oct 30 20:36:59.226606 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:59.226572 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqnvp\" (UniqueName: \"kubernetes.io/projected/25776845-32c3-4c70-aa99-26fe1fa5ecd0-kube-api-access-lqnvp\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-547082-hfgnm\" (UID: \"25776845-32c3-4c70-aa99-26fe1fa5ecd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-547082-hfgnm" Oct 30 20:36:59.307669 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:59.307634 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-547082-hfgnm" Oct 30 20:36:59.681888 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:59.681859 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-547082-hfgnm"] Oct 30 20:36:59.683691 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:36:59.683667 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25776845_32c3_4c70_aa99_26fe1fa5ecd0.slice/crio-0cd9f51ab9e0fbbf8c93b10188a7a64eaa097bc287ae4a141746a485cd54fabc WatchSource:0}: Error finding container 0cd9f51ab9e0fbbf8c93b10188a7a64eaa097bc287ae4a141746a485cd54fabc: Status 404 returned error can't find the container with id 0cd9f51ab9e0fbbf8c93b10188a7a64eaa097bc287ae4a141746a485cd54fabc Oct 30 20:36:59.841157 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:36:59.841123 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-547082-hfgnm" event={"ID":"25776845-32c3-4c70-aa99-26fe1fa5ecd0","Type":"ContainerStarted","Data":"0cd9f51ab9e0fbbf8c93b10188a7a64eaa097bc287ae4a141746a485cd54fabc"} Oct 30 20:37:00.325823 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:00.325791 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-98a19b-779f877d8-hgncx" Oct 30 20:37:00.369128 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:00.369094 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8810ca2-97ba-4a61-84da-fc464be80c17-kube-api-access-rbj5g\" (UniqueName: \"kubernetes.io/projected/e8810ca2-97ba-4a61-84da-fc464be80c17-kube-api-access-rbj5g\") pod \"e8810ca2-97ba-4a61-84da-fc464be80c17\" (UID: \"e8810ca2-97ba-4a61-84da-fc464be80c17\") " Oct 30 20:37:00.369257 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:00.369207 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8810ca2-97ba-4a61-84da-fc464be80c17-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8810ca2-97ba-4a61-84da-fc464be80c17-plugins-config-volume\") pod \"e8810ca2-97ba-4a61-84da-fc464be80c17\" (UID: \"e8810ca2-97ba-4a61-84da-fc464be80c17\") " Oct 30 20:37:00.369622 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:00.369594 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8810ca2-97ba-4a61-84da-fc464be80c17-plugins-config-volume" pod "e8810ca2-97ba-4a61-84da-fc464be80c17" (UID: "e8810ca2-97ba-4a61-84da-fc464be80c17"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:37:00.371405 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:00.371377 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8810ca2-97ba-4a61-84da-fc464be80c17-kube-api-access-rbj5g" pod "e8810ca2-97ba-4a61-84da-fc464be80c17" (UID: "e8810ca2-97ba-4a61-84da-fc464be80c17"). InnerVolumeSpecName "kube-api-access-rbj5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:37:00.470395 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:00.470338 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbj5g\" (UniqueName: \"kubernetes.io/projected/e8810ca2-97ba-4a61-84da-fc464be80c17-kube-api-access-rbj5g\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:37:00.470395 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:00.470397 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8810ca2-97ba-4a61-84da-fc464be80c17-plugins-config-volume\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:37:00.847680 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:00.847588 4292 generic.go:356] "Generic (PLEG): container finished" podID="e8810ca2-97ba-4a61-84da-fc464be80c17" containerID="bfb441cca7276ae3e08df2c77f3f4ec1a6c0a473e4347a1eec4ccee3f07a13b1" exitCode=0 Oct 30 20:37:00.847680 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:00.847641 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-98a19b-779f877d8-hgncx" Oct 30 20:37:00.847917 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:00.847671 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-98a19b-779f877d8-hgncx" event={"ID":"e8810ca2-97ba-4a61-84da-fc464be80c17","Type":"ContainerDied","Data":"bfb441cca7276ae3e08df2c77f3f4ec1a6c0a473e4347a1eec4ccee3f07a13b1"} Oct 30 20:37:00.847917 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:00.847728 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-98a19b-779f877d8-hgncx" event={"ID":"e8810ca2-97ba-4a61-84da-fc464be80c17","Type":"ContainerDied","Data":"854a73591375fbffeb5d5fddb2c9a6c00470b8671a065c7af34007d86fde8fde"} Oct 30 20:37:00.847917 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:00.847755 4292 scope.go:122] "RemoveContainer" containerID="bfb441cca7276ae3e08df2c77f3f4ec1a6c0a473e4347a1eec4ccee3f07a13b1" Oct 30 20:37:00.849594 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:00.849570 4292 generic.go:356] "Generic (PLEG): container finished" podID="25776845-32c3-4c70-aa99-26fe1fa5ecd0" containerID="808b9e688a50fef843b1db9b538b1409682a4bfb7e78454304905df6d4264b4c" exitCode=0 Oct 30 20:37:00.849728 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:00.849643 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-547082-hfgnm" event={"ID":"25776845-32c3-4c70-aa99-26fe1fa5ecd0","Type":"ContainerDied","Data":"808b9e688a50fef843b1db9b538b1409682a4bfb7e78454304905df6d4264b4c"} Oct 30 20:37:00.857076 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:00.857053 4292 scope.go:122] "RemoveContainer" containerID="bfb441cca7276ae3e08df2c77f3f4ec1a6c0a473e4347a1eec4ccee3f07a13b1" Oct 30 20:37:00.857397 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:37:00.857375 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfb441cca7276ae3e08df2c77f3f4ec1a6c0a473e4347a1eec4ccee3f07a13b1\": container with ID starting with bfb441cca7276ae3e08df2c77f3f4ec1a6c0a473e4347a1eec4ccee3f07a13b1 not found: ID does not exist" containerID="bfb441cca7276ae3e08df2c77f3f4ec1a6c0a473e4347a1eec4ccee3f07a13b1" Oct 30 20:37:00.857473 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:00.857409 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfb441cca7276ae3e08df2c77f3f4ec1a6c0a473e4347a1eec4ccee3f07a13b1"} err="failed to get container status \"bfb441cca7276ae3e08df2c77f3f4ec1a6c0a473e4347a1eec4ccee3f07a13b1\": rpc error: code = NotFound desc = could not find container \"bfb441cca7276ae3e08df2c77f3f4ec1a6c0a473e4347a1eec4ccee3f07a13b1\": container with ID starting with bfb441cca7276ae3e08df2c77f3f4ec1a6c0a473e4347a1eec4ccee3f07a13b1 not found: ID does not exist" Oct 30 20:37:01.008637 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:01.008606 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a659f45-f2vdb"] Oct 30 20:37:01.009003 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:01.008992 4292 cpu_manager.go:414] "containerMap: removing container" podUID="e8810ca2-97ba-4a61-84da-fc464be80c17" containerName="payload-processing" Oct 30 20:37:01.009040 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:01.009004 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8810ca2-97ba-4a61-84da-fc464be80c17" containerName="payload-processing" Oct 30 20:37:01.009089 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:01.009083 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8810ca2-97ba-4a61-84da-fc464be80c17" containerName="payload-processing" Oct 30 20:37:01.013487 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:01.013464 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a659f45-f2vdb" Oct 30 20:37:01.021885 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:01.021861 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-98a19b-779f877d8-hgncx"] Oct 30 20:37:01.023785 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:01.023758 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-98a19b-779f877d8-hgncx"] Oct 30 20:37:01.077790 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:01.077762 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fd35199-fa8d-403d-a97e-73de633b5052-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a659f45-f2vdb\" (UID: \"3fd35199-fa8d-403d-a97e-73de633b5052\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a659f45-f2vdb" Oct 30 20:37:01.077920 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:01.077860 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r5w7\" (UniqueName: \"kubernetes.io/projected/3fd35199-fa8d-403d-a97e-73de633b5052-kube-api-access-8r5w7\") pod \"maas-api-revoke-sub-4a659f45-f2vdb\" (UID: \"3fd35199-fa8d-403d-a97e-73de633b5052\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a659f45-f2vdb" Oct 30 20:37:01.077920 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:01.077891 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fd35199-fa8d-403d-a97e-73de633b5052-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a659f45-f2vdb\" (UID: \"3fd35199-fa8d-403d-a97e-73de633b5052\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a659f45-f2vdb" Oct 30 20:37:01.103992 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:01.103930 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a659f45-f2vdb"] Oct 30 20:37:01.116632 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:01.116606 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28cdc84e-kl2cb"] Oct 30 20:37:01.120130 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:01.120110 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28cdc84e-kl2cb" Oct 30 20:37:01.175033 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:01.175007 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28cdc84e-kl2cb"] Oct 30 20:37:01.179071 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:01.179048 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fd35199-fa8d-403d-a97e-73de633b5052-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a659f45-f2vdb\" (UID: \"3fd35199-fa8d-403d-a97e-73de633b5052\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a659f45-f2vdb" Oct 30 20:37:01.179192 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:01.179081 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfmnt\" (UniqueName: \"kubernetes.io/projected/bbb2a8ea-5586-4dcd-a257-385a24859651-kube-api-access-dfmnt\") pod \"maas-api-revoke-sub-28cdc84e-kl2cb\" (UID: \"bbb2a8ea-5586-4dcd-a257-385a24859651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28cdc84e-kl2cb" Oct 30 20:37:01.179192 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:01.179113 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r5w7\" (UniqueName: \"kubernetes.io/projected/3fd35199-fa8d-403d-a97e-73de633b5052-kube-api-access-8r5w7\") pod \"maas-api-revoke-sub-4a659f45-f2vdb\" (UID: \"3fd35199-fa8d-403d-a97e-73de633b5052\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a659f45-f2vdb" Oct 30 20:37:01.179192 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:01.179141 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fd35199-fa8d-403d-a97e-73de633b5052-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a659f45-f2vdb\" (UID: \"3fd35199-fa8d-403d-a97e-73de633b5052\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a659f45-f2vdb" Oct 30 20:37:01.179324 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:01.179224 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbb2a8ea-5586-4dcd-a257-385a24859651-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28cdc84e-kl2cb\" (UID: \"bbb2a8ea-5586-4dcd-a257-385a24859651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28cdc84e-kl2cb" Oct 30 20:37:01.179324 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:01.179313 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbb2a8ea-5586-4dcd-a257-385a24859651-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28cdc84e-kl2cb\" (UID: \"bbb2a8ea-5586-4dcd-a257-385a24859651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28cdc84e-kl2cb" Oct 30 20:37:01.179941 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:01.179922 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fd35199-fa8d-403d-a97e-73de633b5052-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a659f45-f2vdb\" (UID: \"3fd35199-fa8d-403d-a97e-73de633b5052\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a659f45-f2vdb" Oct 30 20:37:01.181824 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:01.181803 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fd35199-fa8d-403d-a97e-73de633b5052-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a659f45-f2vdb\" (UID: \"3fd35199-fa8d-403d-a97e-73de633b5052\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a659f45-f2vdb" Oct 30 20:37:01.242596 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:01.242541 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r5w7\" (UniqueName: \"kubernetes.io/projected/3fd35199-fa8d-403d-a97e-73de633b5052-kube-api-access-8r5w7\") pod \"maas-api-revoke-sub-4a659f45-f2vdb\" (UID: \"3fd35199-fa8d-403d-a97e-73de633b5052\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a659f45-f2vdb" Oct 30 20:37:01.280228 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:01.280199 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfmnt\" (UniqueName: \"kubernetes.io/projected/bbb2a8ea-5586-4dcd-a257-385a24859651-kube-api-access-dfmnt\") pod \"maas-api-revoke-sub-28cdc84e-kl2cb\" (UID: \"bbb2a8ea-5586-4dcd-a257-385a24859651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28cdc84e-kl2cb" Oct 30 20:37:01.280378 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:01.280258 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbb2a8ea-5586-4dcd-a257-385a24859651-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28cdc84e-kl2cb\" (UID: \"bbb2a8ea-5586-4dcd-a257-385a24859651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28cdc84e-kl2cb" Oct 30 20:37:01.280378 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:01.280307 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbb2a8ea-5586-4dcd-a257-385a24859651-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28cdc84e-kl2cb\" (UID: \"bbb2a8ea-5586-4dcd-a257-385a24859651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28cdc84e-kl2cb" Oct 30 20:37:01.280922 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:01.280899 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbb2a8ea-5586-4dcd-a257-385a24859651-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28cdc84e-kl2cb\" (UID: \"bbb2a8ea-5586-4dcd-a257-385a24859651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28cdc84e-kl2cb" Oct 30 20:37:01.282648 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:01.282630 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbb2a8ea-5586-4dcd-a257-385a24859651-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28cdc84e-kl2cb\" (UID: \"bbb2a8ea-5586-4dcd-a257-385a24859651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28cdc84e-kl2cb" Oct 30 20:37:01.327643 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:01.327618 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a659f45-f2vdb" Oct 30 20:37:01.342679 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:01.342659 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfmnt\" (UniqueName: \"kubernetes.io/projected/bbb2a8ea-5586-4dcd-a257-385a24859651-kube-api-access-dfmnt\") pod \"maas-api-revoke-sub-28cdc84e-kl2cb\" (UID: \"bbb2a8ea-5586-4dcd-a257-385a24859651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28cdc84e-kl2cb" Oct 30 20:37:01.433678 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:01.433648 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28cdc84e-kl2cb" Oct 30 20:37:01.703720 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:01.703696 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a659f45-f2vdb"] Oct 30 20:37:01.705246 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:37:01.705219 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fd35199_fa8d_403d_a97e_73de633b5052.slice/crio-329adc01dcf72e816dc923bba1838c70958b109c0c04ef6cef67e354ef1aea0c WatchSource:0}: Error finding container 329adc01dcf72e816dc923bba1838c70958b109c0c04ef6cef67e354ef1aea0c: Status 404 returned error can't find the container with id 329adc01dcf72e816dc923bba1838c70958b109c0c04ef6cef67e354ef1aea0c Oct 30 20:37:01.825441 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:01.825418 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28cdc84e-kl2cb"] Oct 30 20:37:01.826881 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:37:01.826854 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbb2a8ea_5586_4dcd_a257_385a24859651.slice/crio-570ae3f6ab450238cc9d4d9f054ddfe3ef14532af48ef716d1f9d4bd17be32b6 WatchSource:0}: Error finding container 570ae3f6ab450238cc9d4d9f054ddfe3ef14532af48ef716d1f9d4bd17be32b6: Status 404 returned error can't find the container with id 570ae3f6ab450238cc9d4d9f054ddfe3ef14532af48ef716d1f9d4bd17be32b6 Oct 30 20:37:01.857319 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:01.857284 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a659f45-f2vdb" event={"ID":"3fd35199-fa8d-403d-a97e-73de633b5052","Type":"ContainerStarted","Data":"329adc01dcf72e816dc923bba1838c70958b109c0c04ef6cef67e354ef1aea0c"} Oct 30 20:37:01.858784 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:01.858761 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28cdc84e-kl2cb" event={"ID":"bbb2a8ea-5586-4dcd-a257-385a24859651","Type":"ContainerStarted","Data":"570ae3f6ab450238cc9d4d9f054ddfe3ef14532af48ef716d1f9d4bd17be32b6"} Oct 30 20:37:02.034929 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:02.034906 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-547082-hfgnm" Oct 30 20:37:02.090328 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:02.090296 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25776845-32c3-4c70-aa99-26fe1fa5ecd0-kube-api-access-lqnvp\" (UniqueName: \"kubernetes.io/projected/25776845-32c3-4c70-aa99-26fe1fa5ecd0-kube-api-access-lqnvp\") pod \"25776845-32c3-4c70-aa99-26fe1fa5ecd0\" (UID: \"25776845-32c3-4c70-aa99-26fe1fa5ecd0\") " Oct 30 20:37:02.090506 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:02.090492 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25776845-32c3-4c70-aa99-26fe1fa5ecd0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25776845-32c3-4c70-aa99-26fe1fa5ecd0-maas-api-openshift-ca\") pod \"25776845-32c3-4c70-aa99-26fe1fa5ecd0\" (UID: \"25776845-32c3-4c70-aa99-26fe1fa5ecd0\") " Oct 30 20:37:02.090685 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:02.090663 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25776845-32c3-4c70-aa99-26fe1fa5ecd0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25776845-32c3-4c70-aa99-26fe1fa5ecd0-maas-api-xks-ca\") pod \"25776845-32c3-4c70-aa99-26fe1fa5ecd0\" (UID: \"25776845-32c3-4c70-aa99-26fe1fa5ecd0\") " Oct 30 20:37:02.090852 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:02.090831 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25776845-32c3-4c70-aa99-26fe1fa5ecd0-maas-api-openshift-ca" pod "25776845-32c3-4c70-aa99-26fe1fa5ecd0" (UID: "25776845-32c3-4c70-aa99-26fe1fa5ecd0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:37:02.090958 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:02.090943 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25776845-32c3-4c70-aa99-26fe1fa5ecd0-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:37:02.092451 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:02.092433 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25776845-32c3-4c70-aa99-26fe1fa5ecd0-kube-api-access-lqnvp" pod "25776845-32c3-4c70-aa99-26fe1fa5ecd0" (UID: "25776845-32c3-4c70-aa99-26fe1fa5ecd0"). InnerVolumeSpecName "kube-api-access-lqnvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:37:02.092524 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:02.092472 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25776845-32c3-4c70-aa99-26fe1fa5ecd0-maas-api-xks-ca" pod "25776845-32c3-4c70-aa99-26fe1fa5ecd0" (UID: "25776845-32c3-4c70-aa99-26fe1fa5ecd0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:37:02.191957 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:02.191815 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25776845-32c3-4c70-aa99-26fe1fa5ecd0-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:37:02.191957 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:02.191840 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqnvp\" (UniqueName: \"kubernetes.io/projected/25776845-32c3-4c70-aa99-26fe1fa5ecd0-kube-api-access-lqnvp\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:37:02.581724 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:02.581694 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8810ca2-97ba-4a61-84da-fc464be80c17" path="/var/lib/kubelet/pods/e8810ca2-97ba-4a61-84da-fc464be80c17/volumes" Oct 30 20:37:02.866647 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:02.866544 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28cdc84e-kl2cb" event={"ID":"bbb2a8ea-5586-4dcd-a257-385a24859651","Type":"ContainerStarted","Data":"48c79ca44f9975a2107c90da9fbd46728603e223e47e0991014fe4a4c1e57d2f"} Oct 30 20:37:02.868130 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:02.868105 4292 generic.go:356] "Generic (PLEG): container finished" podID="3fd35199-fa8d-403d-a97e-73de633b5052" containerID="2dfe376e748f0eba0083c69cda47a92257d377153a9a6c1945cc36be4770cf5c" exitCode=0 Oct 30 20:37:02.868244 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:02.868171 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a659f45-f2vdb" event={"ID":"3fd35199-fa8d-403d-a97e-73de633b5052","Type":"ContainerDied","Data":"2dfe376e748f0eba0083c69cda47a92257d377153a9a6c1945cc36be4770cf5c"} Oct 30 20:37:02.869444 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:02.869420 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-547082-hfgnm" event={"ID":"25776845-32c3-4c70-aa99-26fe1fa5ecd0","Type":"ContainerDied","Data":"0cd9f51ab9e0fbbf8c93b10188a7a64eaa097bc287ae4a141746a485cd54fabc"} Oct 30 20:37:02.869570 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:02.869446 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-547082-hfgnm" Oct 30 20:37:02.869570 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:02.869467 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0cd9f51ab9e0fbbf8c93b10188a7a64eaa097bc287ae4a141746a485cd54fabc" Oct 30 20:37:02.989109 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:02.989059 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28cdc84e-kl2cb" podStartSLOduration=1.989042338 podStartE2EDuration="1.989042338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:37:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:02.986959407 +0000 UTC m=+4326.505274900" watchObservedRunningTime="2026-10-30 20:37:02.989042338 +0000 UTC m=+4326.507357793" Oct 30 20:37:03.309377 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:03.309312 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms" Oct 30 20:37:03.406686 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:03.406659 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d5226bf-9ab0-496b-b2f5-f0d730554365-istio-token\" (UniqueName: \"kubernetes.io/projected/9d5226bf-9ab0-496b-b2f5-f0d730554365-istio-token\") pod \"9d5226bf-9ab0-496b-b2f5-f0d730554365\" (UID: \"9d5226bf-9ab0-496b-b2f5-f0d730554365\") " Oct 30 20:37:03.406827 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:03.406712 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d5226bf-9ab0-496b-b2f5-f0d730554365-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d5226bf-9ab0-496b-b2f5-f0d730554365-istio-data\") pod \"9d5226bf-9ab0-496b-b2f5-f0d730554365\" (UID: \"9d5226bf-9ab0-496b-b2f5-f0d730554365\") " Oct 30 20:37:03.406827 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:03.406767 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d5226bf-9ab0-496b-b2f5-f0d730554365-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d5226bf-9ab0-496b-b2f5-f0d730554365-workload-socket\") pod \"9d5226bf-9ab0-496b-b2f5-f0d730554365\" (UID: \"9d5226bf-9ab0-496b-b2f5-f0d730554365\") " Oct 30 20:37:03.406827 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:03.406794 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d5226bf-9ab0-496b-b2f5-f0d730554365-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d5226bf-9ab0-496b-b2f5-f0d730554365-istiod-ca-cert\") pod \"9d5226bf-9ab0-496b-b2f5-f0d730554365\" (UID: \"9d5226bf-9ab0-496b-b2f5-f0d730554365\") " Oct 30 20:37:03.406827 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:03.406820 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d5226bf-9ab0-496b-b2f5-f0d730554365-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d5226bf-9ab0-496b-b2f5-f0d730554365-workload-certs\") pod \"9d5226bf-9ab0-496b-b2f5-f0d730554365\" (UID: \"9d5226bf-9ab0-496b-b2f5-f0d730554365\") " Oct 30 20:37:03.407060 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:03.406896 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d5226bf-9ab0-496b-b2f5-f0d730554365-kube-api-access-t7hhw\" (UniqueName: \"kubernetes.io/projected/9d5226bf-9ab0-496b-b2f5-f0d730554365-kube-api-access-t7hhw\") pod \"9d5226bf-9ab0-496b-b2f5-f0d730554365\" (UID: \"9d5226bf-9ab0-496b-b2f5-f0d730554365\") " Oct 30 20:37:03.407060 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:03.406943 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d5226bf-9ab0-496b-b2f5-f0d730554365-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d5226bf-9ab0-496b-b2f5-f0d730554365-credential-socket\") pod \"9d5226bf-9ab0-496b-b2f5-f0d730554365\" (UID: \"9d5226bf-9ab0-496b-b2f5-f0d730554365\") " Oct 30 20:37:03.407060 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:03.406965 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d5226bf-9ab0-496b-b2f5-f0d730554365-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d5226bf-9ab0-496b-b2f5-f0d730554365-istio-podinfo\") pod \"9d5226bf-9ab0-496b-b2f5-f0d730554365\" (UID: \"9d5226bf-9ab0-496b-b2f5-f0d730554365\") " Oct 30 20:37:03.407060 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:03.406989 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d5226bf-9ab0-496b-b2f5-f0d730554365-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d5226bf-9ab0-496b-b2f5-f0d730554365-istio-envoy\") pod \"9d5226bf-9ab0-496b-b2f5-f0d730554365\" (UID: \"9d5226bf-9ab0-496b-b2f5-f0d730554365\") " Oct 30 20:37:03.407255 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:03.407086 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d5226bf-9ab0-496b-b2f5-f0d730554365-istio-data" pod "9d5226bf-9ab0-496b-b2f5-f0d730554365" (UID: "9d5226bf-9ab0-496b-b2f5-f0d730554365"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:37:03.407255 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:03.407194 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d5226bf-9ab0-496b-b2f5-f0d730554365-istiod-ca-cert" pod "9d5226bf-9ab0-496b-b2f5-f0d730554365" (UID: "9d5226bf-9ab0-496b-b2f5-f0d730554365"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:37:03.407255 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:03.407081 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d5226bf-9ab0-496b-b2f5-f0d730554365-workload-socket" pod "9d5226bf-9ab0-496b-b2f5-f0d730554365" (UID: "9d5226bf-9ab0-496b-b2f5-f0d730554365"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:37:03.407427 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:03.407330 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d5226bf-9ab0-496b-b2f5-f0d730554365-credential-socket" pod "9d5226bf-9ab0-496b-b2f5-f0d730554365" (UID: "9d5226bf-9ab0-496b-b2f5-f0d730554365"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:37:03.407548 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:03.407523 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d5226bf-9ab0-496b-b2f5-f0d730554365-istio-data\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:37:03.407548 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:03.407550 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d5226bf-9ab0-496b-b2f5-f0d730554365-workload-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:37:03.407851 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:03.407567 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d5226bf-9ab0-496b-b2f5-f0d730554365-istiod-ca-cert\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:37:03.407851 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:03.407581 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d5226bf-9ab0-496b-b2f5-f0d730554365-credential-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:37:03.407851 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:03.407577 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d5226bf-9ab0-496b-b2f5-f0d730554365-workload-certs" pod "9d5226bf-9ab0-496b-b2f5-f0d730554365" (UID: "9d5226bf-9ab0-496b-b2f5-f0d730554365"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:37:03.409291 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:03.409268 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d5226bf-9ab0-496b-b2f5-f0d730554365-istio-envoy" pod "9d5226bf-9ab0-496b-b2f5-f0d730554365" (UID: "9d5226bf-9ab0-496b-b2f5-f0d730554365"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:37:03.409470 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:03.409436 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d5226bf-9ab0-496b-b2f5-f0d730554365-istio-podinfo" pod "9d5226bf-9ab0-496b-b2f5-f0d730554365" (UID: "9d5226bf-9ab0-496b-b2f5-f0d730554365"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:37:03.409470 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:03.409452 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d5226bf-9ab0-496b-b2f5-f0d730554365-kube-api-access-t7hhw" pod "9d5226bf-9ab0-496b-b2f5-f0d730554365" (UID: "9d5226bf-9ab0-496b-b2f5-f0d730554365"). InnerVolumeSpecName "kube-api-access-t7hhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:37:03.409572 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:03.409518 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d5226bf-9ab0-496b-b2f5-f0d730554365-istio-token" pod "9d5226bf-9ab0-496b-b2f5-f0d730554365" (UID: "9d5226bf-9ab0-496b-b2f5-f0d730554365"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:37:03.508083 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:03.508012 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d5226bf-9ab0-496b-b2f5-f0d730554365-workload-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:37:03.508083 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:03.508040 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7hhw\" (UniqueName: \"kubernetes.io/projected/9d5226bf-9ab0-496b-b2f5-f0d730554365-kube-api-access-t7hhw\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:37:03.508083 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:03.508049 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d5226bf-9ab0-496b-b2f5-f0d730554365-istio-podinfo\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:37:03.508083 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:03.508058 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d5226bf-9ab0-496b-b2f5-f0d730554365-istio-envoy\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:37:03.508083 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:03.508066 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d5226bf-9ab0-496b-b2f5-f0d730554365-istio-token\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:37:03.876089 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:03.876051 4292 generic.go:356] "Generic (PLEG): container finished" podID="bbb2a8ea-5586-4dcd-a257-385a24859651" containerID="48c79ca44f9975a2107c90da9fbd46728603e223e47e0991014fe4a4c1e57d2f" exitCode=0 Oct 30 20:37:03.876563 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:03.876128 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28cdc84e-kl2cb" event={"ID":"bbb2a8ea-5586-4dcd-a257-385a24859651","Type":"ContainerDied","Data":"48c79ca44f9975a2107c90da9fbd46728603e223e47e0991014fe4a4c1e57d2f"} Oct 30 20:37:03.877897 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:03.877875 4292 generic.go:356] "Generic (PLEG): container finished" podID="9d5226bf-9ab0-496b-b2f5-f0d730554365" containerID="41e0a2b5abd2268e6d14c9a1757705f4d392117878465c05d42f78d09b9aeb25" exitCode=0 Oct 30 20:37:03.878036 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:03.877934 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms" Oct 30 20:37:03.878036 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:03.877962 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms" event={"ID":"9d5226bf-9ab0-496b-b2f5-f0d730554365","Type":"ContainerDied","Data":"41e0a2b5abd2268e6d14c9a1757705f4d392117878465c05d42f78d09b9aeb25"} Oct 30 20:37:03.878036 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:03.878007 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms" event={"ID":"9d5226bf-9ab0-496b-b2f5-f0d730554365","Type":"ContainerDied","Data":"d7193d5200163cc2379cd35a216422439f048745a331db8326dc9aab1c1a1dc4"} Oct 30 20:37:03.878036 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:03.878027 4292 scope.go:122] "RemoveContainer" containerID="41e0a2b5abd2268e6d14c9a1757705f4d392117878465c05d42f78d09b9aeb25" Oct 30 20:37:03.888396 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:03.888372 4292 scope.go:122] "RemoveContainer" containerID="ac68c6895d5768fa483ea1155501e73eb48cea63371569daa1ec1260f43328ba" Oct 30 20:37:03.897868 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:03.897846 4292 scope.go:122] "RemoveContainer" containerID="41e0a2b5abd2268e6d14c9a1757705f4d392117878465c05d42f78d09b9aeb25" Oct 30 20:37:03.898178 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:37:03.898155 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41e0a2b5abd2268e6d14c9a1757705f4d392117878465c05d42f78d09b9aeb25\": container with ID starting with 41e0a2b5abd2268e6d14c9a1757705f4d392117878465c05d42f78d09b9aeb25 not found: ID does not exist" containerID="41e0a2b5abd2268e6d14c9a1757705f4d392117878465c05d42f78d09b9aeb25" Oct 30 20:37:03.898247 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:03.898186 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41e0a2b5abd2268e6d14c9a1757705f4d392117878465c05d42f78d09b9aeb25"} err="failed to get container status \"41e0a2b5abd2268e6d14c9a1757705f4d392117878465c05d42f78d09b9aeb25\": rpc error: code = NotFound desc = could not find container \"41e0a2b5abd2268e6d14c9a1757705f4d392117878465c05d42f78d09b9aeb25\": container with ID starting with 41e0a2b5abd2268e6d14c9a1757705f4d392117878465c05d42f78d09b9aeb25 not found: ID does not exist" Oct 30 20:37:03.898247 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:03.898205 4292 scope.go:122] "RemoveContainer" containerID="ac68c6895d5768fa483ea1155501e73eb48cea63371569daa1ec1260f43328ba" Oct 30 20:37:03.898505 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:37:03.898477 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac68c6895d5768fa483ea1155501e73eb48cea63371569daa1ec1260f43328ba\": container with ID starting with ac68c6895d5768fa483ea1155501e73eb48cea63371569daa1ec1260f43328ba not found: ID does not exist" containerID="ac68c6895d5768fa483ea1155501e73eb48cea63371569daa1ec1260f43328ba" Oct 30 20:37:03.898585 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:03.898504 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac68c6895d5768fa483ea1155501e73eb48cea63371569daa1ec1260f43328ba"} err="failed to get container status \"ac68c6895d5768fa483ea1155501e73eb48cea63371569daa1ec1260f43328ba\": rpc error: code = NotFound desc = could not find container \"ac68c6895d5768fa483ea1155501e73eb48cea63371569daa1ec1260f43328ba\": container with ID starting with ac68c6895d5768fa483ea1155501e73eb48cea63371569daa1ec1260f43328ba not found: ID does not exist" Oct 30 20:37:03.997084 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:03.997060 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a659f45-f2vdb" Oct 30 20:37:04.109102 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:04.109069 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms"] Oct 30 20:37:04.113604 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:04.113581 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fd35199-fa8d-403d-a97e-73de633b5052-kube-api-access-8r5w7\" (UniqueName: \"kubernetes.io/projected/3fd35199-fa8d-403d-a97e-73de633b5052-kube-api-access-8r5w7\") pod \"3fd35199-fa8d-403d-a97e-73de633b5052\" (UID: \"3fd35199-fa8d-403d-a97e-73de633b5052\") " Oct 30 20:37:04.113721 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:04.113687 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fd35199-fa8d-403d-a97e-73de633b5052-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fd35199-fa8d-403d-a97e-73de633b5052-maas-api-openshift-ca\") pod \"3fd35199-fa8d-403d-a97e-73de633b5052\" (UID: \"3fd35199-fa8d-403d-a97e-73de633b5052\") " Oct 30 20:37:04.113721 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:04.113715 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fd35199-fa8d-403d-a97e-73de633b5052-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fd35199-fa8d-403d-a97e-73de633b5052-maas-api-xks-ca\") pod \"3fd35199-fa8d-403d-a97e-73de633b5052\" (UID: \"3fd35199-fa8d-403d-a97e-73de633b5052\") " Oct 30 20:37:04.114084 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:04.114056 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fd35199-fa8d-403d-a97e-73de633b5052-maas-api-openshift-ca" pod "3fd35199-fa8d-403d-a97e-73de633b5052" (UID: "3fd35199-fa8d-403d-a97e-73de633b5052"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:37:04.115628 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:04.115610 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fd35199-fa8d-403d-a97e-73de633b5052-maas-api-xks-ca" pod "3fd35199-fa8d-403d-a97e-73de633b5052" (UID: "3fd35199-fa8d-403d-a97e-73de633b5052"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:37:04.115734 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:04.115719 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fd35199-fa8d-403d-a97e-73de633b5052-kube-api-access-8r5w7" pod "3fd35199-fa8d-403d-a97e-73de633b5052" (UID: "3fd35199-fa8d-403d-a97e-73de633b5052"). InnerVolumeSpecName "kube-api-access-8r5w7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:37:04.159867 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:04.159802 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-e391de-openshift-default-55f885987c-qs9ms"] Oct 30 20:37:04.215115 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:04.215071 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fd35199-fa8d-403d-a97e-73de633b5052-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:37:04.215115 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:04.215107 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fd35199-fa8d-403d-a97e-73de633b5052-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:37:04.215115 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:04.215121 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r5w7\" (UniqueName: \"kubernetes.io/projected/3fd35199-fa8d-403d-a97e-73de633b5052-kube-api-access-8r5w7\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:37:04.581855 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:04.581822 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d5226bf-9ab0-496b-b2f5-f0d730554365" path="/var/lib/kubelet/pods/9d5226bf-9ab0-496b-b2f5-f0d730554365/volumes" Oct 30 20:37:04.886219 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:04.886131 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a659f45-f2vdb" Oct 30 20:37:04.886219 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:04.886173 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a659f45-f2vdb" event={"ID":"3fd35199-fa8d-403d-a97e-73de633b5052","Type":"ContainerDied","Data":"329adc01dcf72e816dc923bba1838c70958b109c0c04ef6cef67e354ef1aea0c"} Oct 30 20:37:04.886709 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:04.886232 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="329adc01dcf72e816dc923bba1838c70958b109c0c04ef6cef67e354ef1aea0c" Oct 30 20:37:05.018740 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:05.018712 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28cdc84e-kl2cb" Oct 30 20:37:05.123282 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:05.123251 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbb2a8ea-5586-4dcd-a257-385a24859651-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbb2a8ea-5586-4dcd-a257-385a24859651-maas-api-openshift-ca\") pod \"bbb2a8ea-5586-4dcd-a257-385a24859651\" (UID: \"bbb2a8ea-5586-4dcd-a257-385a24859651\") " Oct 30 20:37:05.123440 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:05.123326 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbb2a8ea-5586-4dcd-a257-385a24859651-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbb2a8ea-5586-4dcd-a257-385a24859651-maas-api-xks-ca\") pod \"bbb2a8ea-5586-4dcd-a257-385a24859651\" (UID: \"bbb2a8ea-5586-4dcd-a257-385a24859651\") " Oct 30 20:37:05.123505 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:05.123484 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbb2a8ea-5586-4dcd-a257-385a24859651-kube-api-access-dfmnt\" (UniqueName: \"kubernetes.io/projected/bbb2a8ea-5586-4dcd-a257-385a24859651-kube-api-access-dfmnt\") pod \"bbb2a8ea-5586-4dcd-a257-385a24859651\" (UID: \"bbb2a8ea-5586-4dcd-a257-385a24859651\") " Oct 30 20:37:05.123632 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:05.123609 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbb2a8ea-5586-4dcd-a257-385a24859651-maas-api-openshift-ca" pod "bbb2a8ea-5586-4dcd-a257-385a24859651" (UID: "bbb2a8ea-5586-4dcd-a257-385a24859651"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:37:05.123838 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:05.123817 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbb2a8ea-5586-4dcd-a257-385a24859651-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:37:05.125272 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:05.125254 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbb2a8ea-5586-4dcd-a257-385a24859651-maas-api-xks-ca" pod "bbb2a8ea-5586-4dcd-a257-385a24859651" (UID: "bbb2a8ea-5586-4dcd-a257-385a24859651"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:37:05.125472 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:05.125453 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbb2a8ea-5586-4dcd-a257-385a24859651-kube-api-access-dfmnt" pod "bbb2a8ea-5586-4dcd-a257-385a24859651" (UID: "bbb2a8ea-5586-4dcd-a257-385a24859651"). InnerVolumeSpecName "kube-api-access-dfmnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:37:05.224714 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:05.224638 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfmnt\" (UniqueName: \"kubernetes.io/projected/bbb2a8ea-5586-4dcd-a257-385a24859651-kube-api-access-dfmnt\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:37:05.224714 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:05.224667 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbb2a8ea-5586-4dcd-a257-385a24859651-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:37:05.892796 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:05.892770 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28cdc84e-kl2cb" Oct 30 20:37:05.892796 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:05.892776 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28cdc84e-kl2cb" event={"ID":"bbb2a8ea-5586-4dcd-a257-385a24859651","Type":"ContainerDied","Data":"570ae3f6ab450238cc9d4d9f054ddfe3ef14532af48ef716d1f9d4bd17be32b6"} Oct 30 20:37:05.893206 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:05.892824 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="570ae3f6ab450238cc9d4d9f054ddfe3ef14532af48ef716d1f9d4bd17be32b6" Oct 30 20:37:13.219417 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:37:13.219373 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:37:27.262477 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:27.262447 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-273c1b-787f5dc9d4-pt68w"] Oct 30 20:37:27.262957 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:27.262817 4292 cpu_manager.go:414] "containerMap: removing container" podUID="9d5226bf-9ab0-496b-b2f5-f0d730554365" containerName="istio-proxy" Oct 30 20:37:27.262957 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:27.262829 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d5226bf-9ab0-496b-b2f5-f0d730554365" containerName="istio-proxy" Oct 30 20:37:27.262957 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:27.262860 4292 cpu_manager.go:414] "containerMap: removing container" podUID="bbb2a8ea-5586-4dcd-a257-385a24859651" containerName="revoke-keys" Oct 30 20:37:27.262957 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:27.262865 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbb2a8ea-5586-4dcd-a257-385a24859651" containerName="revoke-keys" Oct 30 20:37:27.262957 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:27.262873 4292 cpu_manager.go:414] "containerMap: removing container" podUID="3fd35199-fa8d-403d-a97e-73de633b5052" containerName="revoke-keys" Oct 30 20:37:27.262957 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:27.262878 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fd35199-fa8d-403d-a97e-73de633b5052" containerName="revoke-keys" Oct 30 20:37:27.262957 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:27.262884 4292 cpu_manager.go:414] "containerMap: removing container" podUID="25776845-32c3-4c70-aa99-26fe1fa5ecd0" containerName="revoke-keys" Oct 30 20:37:27.262957 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:27.262888 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25776845-32c3-4c70-aa99-26fe1fa5ecd0" containerName="revoke-keys" Oct 30 20:37:27.262957 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:27.262938 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d5226bf-9ab0-496b-b2f5-f0d730554365" containerName="istio-proxy" Oct 30 20:37:27.262957 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:27.262957 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbb2a8ea-5586-4dcd-a257-385a24859651" containerName="revoke-keys" Oct 30 20:37:27.262957 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:27.262963 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fd35199-fa8d-403d-a97e-73de633b5052" containerName="revoke-keys" Oct 30 20:37:27.263315 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:27.262968 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="25776845-32c3-4c70-aa99-26fe1fa5ecd0" containerName="revoke-keys" Oct 30 20:37:27.266484 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:27.266465 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-273c1b-787f5dc9d4-pt68w" Oct 30 20:37:27.353090 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:27.353057 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds7ld\" (UniqueName: \"kubernetes.io/projected/6f481086-d4fb-403b-822c-bf182959c81f-kube-api-access-ds7ld\") pod \"maas-api-e2e-extmod-b-273c1b-787f5dc9d4-pt68w\" (UID: \"6f481086-d4fb-403b-822c-bf182959c81f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-273c1b-787f5dc9d4-pt68w" Oct 30 20:37:27.353267 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:27.353105 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f481086-d4fb-403b-822c-bf182959c81f-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-273c1b-787f5dc9d4-pt68w\" (UID: \"6f481086-d4fb-403b-822c-bf182959c81f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-273c1b-787f5dc9d4-pt68w" Oct 30 20:37:27.353267 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:27.353237 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f481086-d4fb-403b-822c-bf182959c81f-metrics-certs\") pod \"maas-api-e2e-extmod-b-273c1b-787f5dc9d4-pt68w\" (UID: \"6f481086-d4fb-403b-822c-bf182959c81f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-273c1b-787f5dc9d4-pt68w" Oct 30 20:37:27.353402 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:27.353335 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f481086-d4fb-403b-822c-bf182959c81f-maas-api-tls\") pod \"maas-api-e2e-extmod-b-273c1b-787f5dc9d4-pt68w\" (UID: \"6f481086-d4fb-403b-822c-bf182959c81f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-273c1b-787f5dc9d4-pt68w" Oct 30 20:37:27.371146 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:27.371119 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-273c1b-serving-cert\"" Oct 30 20:37:27.373483 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:27.373458 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-273c1b-787f5dc9d4-pt68w"] Oct 30 20:37:27.454287 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:27.454256 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds7ld\" (UniqueName: \"kubernetes.io/projected/6f481086-d4fb-403b-822c-bf182959c81f-kube-api-access-ds7ld\") pod \"maas-api-e2e-extmod-b-273c1b-787f5dc9d4-pt68w\" (UID: \"6f481086-d4fb-403b-822c-bf182959c81f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-273c1b-787f5dc9d4-pt68w" Oct 30 20:37:27.454287 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:27.454293 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f481086-d4fb-403b-822c-bf182959c81f-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-273c1b-787f5dc9d4-pt68w\" (UID: \"6f481086-d4fb-403b-822c-bf182959c81f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-273c1b-787f5dc9d4-pt68w" Oct 30 20:37:27.454516 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:27.454419 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f481086-d4fb-403b-822c-bf182959c81f-metrics-certs\") pod \"maas-api-e2e-extmod-b-273c1b-787f5dc9d4-pt68w\" (UID: \"6f481086-d4fb-403b-822c-bf182959c81f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-273c1b-787f5dc9d4-pt68w" Oct 30 20:37:27.454516 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:27.454473 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f481086-d4fb-403b-822c-bf182959c81f-maas-api-tls\") pod \"maas-api-e2e-extmod-b-273c1b-787f5dc9d4-pt68w\" (UID: \"6f481086-d4fb-403b-822c-bf182959c81f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-273c1b-787f5dc9d4-pt68w" Oct 30 20:37:27.454633 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:37:27.454620 4292 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-273c1b-serving-cert: secret "maas-api-e2e-extmod-b-273c1b-serving-cert" not found Oct 30 20:37:27.454694 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:37:27.454684 4292 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f481086-d4fb-403b-822c-bf182959c81f-maas-api-tls podName:6f481086-d4fb-403b-822c-bf182959c81f nodeName:}" failed. No retries permitted until 2026-10-30 20:37:27.954666336 +0000 UTC m=+4351.472981783 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6f481086-d4fb-403b-822c-bf182959c81f-maas-api-tls") pod "maas-api-e2e-extmod-b-273c1b-787f5dc9d4-pt68w" (UID: "6f481086-d4fb-403b-822c-bf182959c81f") : secret "maas-api-e2e-extmod-b-273c1b-serving-cert" not found Oct 30 20:37:27.454949 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:27.454931 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f481086-d4fb-403b-822c-bf182959c81f-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-273c1b-787f5dc9d4-pt68w\" (UID: \"6f481086-d4fb-403b-822c-bf182959c81f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-273c1b-787f5dc9d4-pt68w" Oct 30 20:37:27.456760 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:27.456740 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f481086-d4fb-403b-822c-bf182959c81f-metrics-certs\") pod \"maas-api-e2e-extmod-b-273c1b-787f5dc9d4-pt68w\" (UID: \"6f481086-d4fb-403b-822c-bf182959c81f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-273c1b-787f5dc9d4-pt68w" Oct 30 20:37:27.516991 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:27.516964 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds7ld\" (UniqueName: \"kubernetes.io/projected/6f481086-d4fb-403b-822c-bf182959c81f-kube-api-access-ds7ld\") pod \"maas-api-e2e-extmod-b-273c1b-787f5dc9d4-pt68w\" (UID: \"6f481086-d4fb-403b-822c-bf182959c81f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-273c1b-787f5dc9d4-pt68w" Oct 30 20:37:27.964556 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:27.964466 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f481086-d4fb-403b-822c-bf182959c81f-maas-api-tls\") pod \"maas-api-e2e-extmod-b-273c1b-787f5dc9d4-pt68w\" (UID: \"6f481086-d4fb-403b-822c-bf182959c81f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-273c1b-787f5dc9d4-pt68w" Oct 30 20:37:27.967081 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:27.967049 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f481086-d4fb-403b-822c-bf182959c81f-maas-api-tls\") pod \"maas-api-e2e-extmod-b-273c1b-787f5dc9d4-pt68w\" (UID: \"6f481086-d4fb-403b-822c-bf182959c81f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-273c1b-787f5dc9d4-pt68w" Oct 30 20:37:28.178183 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:28.178156 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-273c1b-787f5dc9d4-pt68w" Oct 30 20:37:28.554818 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:28.554791 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-273c1b-787f5dc9d4-pt68w"] Oct 30 20:37:28.556113 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:37:28.556079 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f481086_d4fb_403b_822c_bf182959c81f.slice/crio-877e373a53f0fb7303be35213fa29c8f1d59dfd4220c6d5ac8277e0b535d2c4c WatchSource:0}: Error finding container 877e373a53f0fb7303be35213fa29c8f1d59dfd4220c6d5ac8277e0b535d2c4c: Status 404 returned error can't find the container with id 877e373a53f0fb7303be35213fa29c8f1d59dfd4220c6d5ac8277e0b535d2c4c Oct 30 20:37:28.557556 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:28.557535 4292 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 20:37:29.013761 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:29.013674 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-273c1b-787f5dc9d4-pt68w" event={"ID":"6f481086-d4fb-403b-822c-bf182959c81f","Type":"ContainerStarted","Data":"877e373a53f0fb7303be35213fa29c8f1d59dfd4220c6d5ac8277e0b535d2c4c"} Oct 30 20:37:30.020803 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:30.020769 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-273c1b-787f5dc9d4-pt68w" event={"ID":"6f481086-d4fb-403b-822c-bf182959c81f","Type":"ContainerStarted","Data":"d60512a22e5a14c8a1d2b383079585a6178fb0b90b6f48bef47c1a2339c16c87"} Oct 30 20:37:30.021208 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:30.020836 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-273c1b-787f5dc9d4-pt68w" Oct 30 20:37:30.141855 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:30.141792 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-273c1b-787f5dc9d4-pt68w" podStartSLOduration=2.152044853 podStartE2EDuration="3.141776248s" totalImagesPullingTime="989.731395ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:37:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:30.140780413 +0000 UTC m=+4353.659095869" watchObservedRunningTime="2026-10-30 20:37:30.141776248 +0000 UTC m=+4353.660091706" Oct 30 20:37:32.044712 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:32.044682 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ec609-754d6f868c-l66gq"] Oct 30 20:37:32.045147 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:32.044869 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ec609-754d6f868c-l66gq" podUID="85408ab0-d248-46df-a359-6a213c10fd58" containerName="maas-api" containerID="cri-o://6a851a32d7202183836dae26f5a0ab8070b798da7ea8eb8108aa75760d05350b" gracePeriod=30 Oct 30 20:37:32.322267 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:32.322233 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ec609-754d6f868c-l66gq" Oct 30 20:37:32.406172 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:32.406131 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85408ab0-d248-46df-a359-6a213c10fd58-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85408ab0-d248-46df-a359-6a213c10fd58-openshift-service-ca\") pod \"85408ab0-d248-46df-a359-6a213c10fd58\" (UID: \"85408ab0-d248-46df-a359-6a213c10fd58\") " Oct 30 20:37:32.406375 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:32.406188 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85408ab0-d248-46df-a359-6a213c10fd58-kube-api-access-q4r5l\" (UniqueName: \"kubernetes.io/projected/85408ab0-d248-46df-a359-6a213c10fd58-kube-api-access-q4r5l\") pod \"85408ab0-d248-46df-a359-6a213c10fd58\" (UID: \"85408ab0-d248-46df-a359-6a213c10fd58\") " Oct 30 20:37:32.406375 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:32.406259 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85408ab0-d248-46df-a359-6a213c10fd58-metrics-certs\" (UniqueName: \"kubernetes.io/secret/85408ab0-d248-46df-a359-6a213c10fd58-metrics-certs\") pod \"85408ab0-d248-46df-a359-6a213c10fd58\" (UID: \"85408ab0-d248-46df-a359-6a213c10fd58\") " Oct 30 20:37:32.406375 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:32.406334 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85408ab0-d248-46df-a359-6a213c10fd58-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85408ab0-d248-46df-a359-6a213c10fd58-maas-api-tls\") pod \"85408ab0-d248-46df-a359-6a213c10fd58\" (UID: \"85408ab0-d248-46df-a359-6a213c10fd58\") " Oct 30 20:37:32.406619 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:32.406598 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85408ab0-d248-46df-a359-6a213c10fd58-openshift-service-ca" pod "85408ab0-d248-46df-a359-6a213c10fd58" (UID: "85408ab0-d248-46df-a359-6a213c10fd58"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:37:32.406710 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:32.406696 4292 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85408ab0-d248-46df-a359-6a213c10fd58-openshift-service-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:37:32.408456 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:32.408430 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85408ab0-d248-46df-a359-6a213c10fd58-metrics-certs" pod "85408ab0-d248-46df-a359-6a213c10fd58" (UID: "85408ab0-d248-46df-a359-6a213c10fd58"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:37:32.408561 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:32.408457 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85408ab0-d248-46df-a359-6a213c10fd58-maas-api-tls" pod "85408ab0-d248-46df-a359-6a213c10fd58" (UID: "85408ab0-d248-46df-a359-6a213c10fd58"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:37:32.408561 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:32.408485 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85408ab0-d248-46df-a359-6a213c10fd58-kube-api-access-q4r5l" pod "85408ab0-d248-46df-a359-6a213c10fd58" (UID: "85408ab0-d248-46df-a359-6a213c10fd58"). InnerVolumeSpecName "kube-api-access-q4r5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:37:32.508136 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:32.508106 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85408ab0-d248-46df-a359-6a213c10fd58-maas-api-tls\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:37:32.508136 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:32.508134 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4r5l\" (UniqueName: \"kubernetes.io/projected/85408ab0-d248-46df-a359-6a213c10fd58-kube-api-access-q4r5l\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:37:32.508358 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:32.508146 4292 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85408ab0-d248-46df-a359-6a213c10fd58-metrics-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:37:33.042526 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:33.042488 4292 generic.go:356] "Generic (PLEG): container finished" podID="85408ab0-d248-46df-a359-6a213c10fd58" containerID="6a851a32d7202183836dae26f5a0ab8070b798da7ea8eb8108aa75760d05350b" exitCode=0 Oct 30 20:37:33.042758 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:33.042563 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ec609-754d6f868c-l66gq" Oct 30 20:37:33.042758 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:33.042603 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ec609-754d6f868c-l66gq" event={"ID":"85408ab0-d248-46df-a359-6a213c10fd58","Type":"ContainerDied","Data":"6a851a32d7202183836dae26f5a0ab8070b798da7ea8eb8108aa75760d05350b"} Oct 30 20:37:33.042758 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:33.042650 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ec609-754d6f868c-l66gq" event={"ID":"85408ab0-d248-46df-a359-6a213c10fd58","Type":"ContainerDied","Data":"fcf2f3ed0eb87f68570b110acb3b6ba91b59f1976b77f75b6d79ddcfdf5b4af6"} Oct 30 20:37:33.042758 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:33.042681 4292 scope.go:122] "RemoveContainer" containerID="6a851a32d7202183836dae26f5a0ab8070b798da7ea8eb8108aa75760d05350b" Oct 30 20:37:33.056815 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:33.056417 4292 scope.go:122] "RemoveContainer" containerID="6a851a32d7202183836dae26f5a0ab8070b798da7ea8eb8108aa75760d05350b" Oct 30 20:37:33.057564 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:37:33.057526 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a851a32d7202183836dae26f5a0ab8070b798da7ea8eb8108aa75760d05350b\": container with ID starting with 6a851a32d7202183836dae26f5a0ab8070b798da7ea8eb8108aa75760d05350b not found: ID does not exist" containerID="6a851a32d7202183836dae26f5a0ab8070b798da7ea8eb8108aa75760d05350b" Oct 30 20:37:33.057709 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:33.057571 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a851a32d7202183836dae26f5a0ab8070b798da7ea8eb8108aa75760d05350b"} err="failed to get container status \"6a851a32d7202183836dae26f5a0ab8070b798da7ea8eb8108aa75760d05350b\": rpc error: code = NotFound desc = could not find container \"6a851a32d7202183836dae26f5a0ab8070b798da7ea8eb8108aa75760d05350b\": container with ID starting with 6a851a32d7202183836dae26f5a0ab8070b798da7ea8eb8108aa75760d05350b not found: ID does not exist" Oct 30 20:37:33.213200 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:33.213163 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ec609-754d6f868c-l66gq"] Oct 30 20:37:33.264214 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:33.264181 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ec609-754d6f868c-l66gq"] Oct 30 20:37:34.334775 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:34.334740 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4ec609-prnk6"] Oct 30 20:37:34.385840 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:34.385812 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4ec609-prnk6"] Oct 30 20:37:34.581988 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:34.581956 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6df184b0-58c1-445b-bc48-9146f0075f31" path="/var/lib/kubelet/pods/6df184b0-58c1-445b-bc48-9146f0075f31/volumes" Oct 30 20:37:34.582375 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:34.582346 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85408ab0-d248-46df-a359-6a213c10fd58" path="/var/lib/kubelet/pods/85408ab0-d248-46df-a359-6a213c10fd58/volumes" Oct 30 20:37:36.031180 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:36.031152 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-273c1b-787f5dc9d4-pt68w" Oct 30 20:37:41.098297 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:41.098204 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bbfa07-qgnvt"] Oct 30 20:37:41.148642 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:41.148614 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bbfa07-qgnvt"] Oct 30 20:37:41.490877 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:41.490773 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4"] Oct 30 20:37:41.491028 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:41.490988 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4" podUID="6d8780e4-df03-426f-8981-f355e0430b39" containerName="istio-proxy" containerID="cri-o://249c54f8a9b0e31bc04c97a3b9318efdbb84fb88a517903daaff589c964d4167" gracePeriod=30 Oct 30 20:37:42.580873 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:42.580841 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0de01d4b-467d-4d84-8f5d-aca0db0fba42" path="/var/lib/kubelet/pods/0de01d4b-467d-4d84-8f5d-aca0db0fba42/volumes" Oct 30 20:37:43.220314 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:37:43.220278 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:37:46.731580 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:46.731552 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4" Oct 30 20:37:46.851288 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:46.851254 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d8780e4-df03-426f-8981-f355e0430b39-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d8780e4-df03-426f-8981-f355e0430b39-istiod-ca-cert\") pod \"6d8780e4-df03-426f-8981-f355e0430b39\" (UID: \"6d8780e4-df03-426f-8981-f355e0430b39\") " Oct 30 20:37:46.851504 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:46.851312 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d8780e4-df03-426f-8981-f355e0430b39-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d8780e4-df03-426f-8981-f355e0430b39-istio-podinfo\") pod \"6d8780e4-df03-426f-8981-f355e0430b39\" (UID: \"6d8780e4-df03-426f-8981-f355e0430b39\") " Oct 30 20:37:46.851504 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:46.851343 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d8780e4-df03-426f-8981-f355e0430b39-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d8780e4-df03-426f-8981-f355e0430b39-workload-certs\") pod \"6d8780e4-df03-426f-8981-f355e0430b39\" (UID: \"6d8780e4-df03-426f-8981-f355e0430b39\") " Oct 30 20:37:46.851504 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:46.851412 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d8780e4-df03-426f-8981-f355e0430b39-istio-token\" (UniqueName: \"kubernetes.io/projected/6d8780e4-df03-426f-8981-f355e0430b39-istio-token\") pod \"6d8780e4-df03-426f-8981-f355e0430b39\" (UID: \"6d8780e4-df03-426f-8981-f355e0430b39\") " Oct 30 20:37:46.851504 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:46.851447 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d8780e4-df03-426f-8981-f355e0430b39-kube-api-access-p9kr7\" (UniqueName: \"kubernetes.io/projected/6d8780e4-df03-426f-8981-f355e0430b39-kube-api-access-p9kr7\") pod \"6d8780e4-df03-426f-8981-f355e0430b39\" (UID: \"6d8780e4-df03-426f-8981-f355e0430b39\") " Oct 30 20:37:46.851504 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:46.851473 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d8780e4-df03-426f-8981-f355e0430b39-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d8780e4-df03-426f-8981-f355e0430b39-istio-envoy\") pod \"6d8780e4-df03-426f-8981-f355e0430b39\" (UID: \"6d8780e4-df03-426f-8981-f355e0430b39\") " Oct 30 20:37:46.851783 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:46.851598 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d8780e4-df03-426f-8981-f355e0430b39-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d8780e4-df03-426f-8981-f355e0430b39-credential-socket\") pod \"6d8780e4-df03-426f-8981-f355e0430b39\" (UID: \"6d8780e4-df03-426f-8981-f355e0430b39\") " Oct 30 20:37:46.851783 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:46.851637 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d8780e4-df03-426f-8981-f355e0430b39-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d8780e4-df03-426f-8981-f355e0430b39-istio-data\") pod \"6d8780e4-df03-426f-8981-f355e0430b39\" (UID: \"6d8780e4-df03-426f-8981-f355e0430b39\") " Oct 30 20:37:46.851783 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:46.851702 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d8780e4-df03-426f-8981-f355e0430b39-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d8780e4-df03-426f-8981-f355e0430b39-workload-socket\") pod \"6d8780e4-df03-426f-8981-f355e0430b39\" (UID: \"6d8780e4-df03-426f-8981-f355e0430b39\") " Oct 30 20:37:46.851783 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:46.851743 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d8780e4-df03-426f-8981-f355e0430b39-workload-certs" pod "6d8780e4-df03-426f-8981-f355e0430b39" (UID: "6d8780e4-df03-426f-8981-f355e0430b39"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:37:46.851783 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:46.851769 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d8780e4-df03-426f-8981-f355e0430b39-istiod-ca-cert" pod "6d8780e4-df03-426f-8981-f355e0430b39" (UID: "6d8780e4-df03-426f-8981-f355e0430b39"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:37:46.852038 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:46.851952 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d8780e4-df03-426f-8981-f355e0430b39-istio-data" pod "6d8780e4-df03-426f-8981-f355e0430b39" (UID: "6d8780e4-df03-426f-8981-f355e0430b39"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:37:46.852088 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:46.852053 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d8780e4-df03-426f-8981-f355e0430b39-credential-socket" pod "6d8780e4-df03-426f-8981-f355e0430b39" (UID: "6d8780e4-df03-426f-8981-f355e0430b39"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:37:46.852088 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:46.852060 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d8780e4-df03-426f-8981-f355e0430b39-istiod-ca-cert\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:37:46.852164 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:46.852088 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d8780e4-df03-426f-8981-f355e0430b39-workload-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:37:46.852164 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:46.852103 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d8780e4-df03-426f-8981-f355e0430b39-istio-data\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:37:46.852259 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:46.852228 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d8780e4-df03-426f-8981-f355e0430b39-workload-socket" pod "6d8780e4-df03-426f-8981-f355e0430b39" (UID: "6d8780e4-df03-426f-8981-f355e0430b39"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:37:46.854032 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:46.854004 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d8780e4-df03-426f-8981-f355e0430b39-kube-api-access-p9kr7" pod "6d8780e4-df03-426f-8981-f355e0430b39" (UID: "6d8780e4-df03-426f-8981-f355e0430b39"). InnerVolumeSpecName "kube-api-access-p9kr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:37:46.854032 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:46.854009 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d8780e4-df03-426f-8981-f355e0430b39-istio-podinfo" pod "6d8780e4-df03-426f-8981-f355e0430b39" (UID: "6d8780e4-df03-426f-8981-f355e0430b39"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:37:46.854223 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:46.854103 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d8780e4-df03-426f-8981-f355e0430b39-istio-token" pod "6d8780e4-df03-426f-8981-f355e0430b39" (UID: "6d8780e4-df03-426f-8981-f355e0430b39"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:37:46.854223 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:46.854134 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d8780e4-df03-426f-8981-f355e0430b39-istio-envoy" pod "6d8780e4-df03-426f-8981-f355e0430b39" (UID: "6d8780e4-df03-426f-8981-f355e0430b39"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:37:46.952751 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:46.952716 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d8780e4-df03-426f-8981-f355e0430b39-istio-podinfo\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:37:46.952751 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:46.952742 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d8780e4-df03-426f-8981-f355e0430b39-istio-token\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:37:46.952751 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:46.952752 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9kr7\" (UniqueName: \"kubernetes.io/projected/6d8780e4-df03-426f-8981-f355e0430b39-kube-api-access-p9kr7\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:37:46.953083 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:46.952763 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d8780e4-df03-426f-8981-f355e0430b39-istio-envoy\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:37:46.953083 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:46.952772 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d8780e4-df03-426f-8981-f355e0430b39-credential-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:37:46.953083 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:46.952780 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d8780e4-df03-426f-8981-f355e0430b39-workload-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:37:47.123610 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:47.123515 4292 generic.go:356] "Generic (PLEG): container finished" podID="6d8780e4-df03-426f-8981-f355e0430b39" containerID="249c54f8a9b0e31bc04c97a3b9318efdbb84fb88a517903daaff589c964d4167" exitCode=0 Oct 30 20:37:47.123610 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:47.123576 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4" Oct 30 20:37:47.123842 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:47.123608 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4" event={"ID":"6d8780e4-df03-426f-8981-f355e0430b39","Type":"ContainerDied","Data":"249c54f8a9b0e31bc04c97a3b9318efdbb84fb88a517903daaff589c964d4167"} Oct 30 20:37:47.123842 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:47.123673 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4" event={"ID":"6d8780e4-df03-426f-8981-f355e0430b39","Type":"ContainerDied","Data":"1c3748213065706173b84e157163efa2c9e26241f4285bded7537a3bfca0baf0"} Oct 30 20:37:47.123842 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:47.123699 4292 scope.go:122] "RemoveContainer" containerID="249c54f8a9b0e31bc04c97a3b9318efdbb84fb88a517903daaff589c964d4167" Oct 30 20:37:47.133053 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:47.133036 4292 scope.go:122] "RemoveContainer" containerID="249c54f8a9b0e31bc04c97a3b9318efdbb84fb88a517903daaff589c964d4167" Oct 30 20:37:47.133327 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:37:47.133310 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"249c54f8a9b0e31bc04c97a3b9318efdbb84fb88a517903daaff589c964d4167\": container with ID starting with 249c54f8a9b0e31bc04c97a3b9318efdbb84fb88a517903daaff589c964d4167 not found: ID does not exist" containerID="249c54f8a9b0e31bc04c97a3b9318efdbb84fb88a517903daaff589c964d4167" Oct 30 20:37:47.133476 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:47.133368 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"249c54f8a9b0e31bc04c97a3b9318efdbb84fb88a517903daaff589c964d4167"} err="failed to get container status \"249c54f8a9b0e31bc04c97a3b9318efdbb84fb88a517903daaff589c964d4167\": rpc error: code = NotFound desc = could not find container \"249c54f8a9b0e31bc04c97a3b9318efdbb84fb88a517903daaff589c964d4167\": container with ID starting with 249c54f8a9b0e31bc04c97a3b9318efdbb84fb88a517903daaff589c964d4167 not found: ID does not exist" Oct 30 20:37:47.298167 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:47.298134 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4"] Oct 30 20:37:47.299974 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:47.299954 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-2bcq4"] Oct 30 20:37:48.581176 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:48.581146 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d8780e4-df03-426f-8981-f355e0430b39" path="/var/lib/kubelet/pods/6d8780e4-df03-426f-8981-f355e0430b39/volumes" Oct 30 20:37:51.161008 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:51.160976 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-98a19b-9477d9fdc-b2sls"] Oct 30 20:37:51.161490 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:51.161170 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-98a19b-9477d9fdc-b2sls" podUID="f7d43882-57a4-43f7-9ad9-c5174007f344" containerName="maas-api" containerID="cri-o://b01f2cfbc155b30f4c01717cbdabc03df2223ce2e9b6a338ac525bbc06e43052" gracePeriod=30 Oct 30 20:37:51.428481 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:51.428455 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-98a19b-9477d9fdc-b2sls" Oct 30 20:37:51.598527 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:51.598500 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7d43882-57a4-43f7-9ad9-c5174007f344-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7d43882-57a4-43f7-9ad9-c5174007f344-maas-api-tls\") pod \"f7d43882-57a4-43f7-9ad9-c5174007f344\" (UID: \"f7d43882-57a4-43f7-9ad9-c5174007f344\") " Oct 30 20:37:51.598701 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:51.598570 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7d43882-57a4-43f7-9ad9-c5174007f344-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7d43882-57a4-43f7-9ad9-c5174007f344-metrics-certs\") pod \"f7d43882-57a4-43f7-9ad9-c5174007f344\" (UID: \"f7d43882-57a4-43f7-9ad9-c5174007f344\") " Oct 30 20:37:51.598758 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:51.598738 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7d43882-57a4-43f7-9ad9-c5174007f344-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7d43882-57a4-43f7-9ad9-c5174007f344-openshift-service-ca\") pod \"f7d43882-57a4-43f7-9ad9-c5174007f344\" (UID: \"f7d43882-57a4-43f7-9ad9-c5174007f344\") " Oct 30 20:37:51.598841 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:51.598825 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7d43882-57a4-43f7-9ad9-c5174007f344-kube-api-access-bfb77\" (UniqueName: \"kubernetes.io/projected/f7d43882-57a4-43f7-9ad9-c5174007f344-kube-api-access-bfb77\") pod \"f7d43882-57a4-43f7-9ad9-c5174007f344\" (UID: \"f7d43882-57a4-43f7-9ad9-c5174007f344\") " Oct 30 20:37:51.599128 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:51.599098 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7d43882-57a4-43f7-9ad9-c5174007f344-openshift-service-ca" pod "f7d43882-57a4-43f7-9ad9-c5174007f344" (UID: "f7d43882-57a4-43f7-9ad9-c5174007f344"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:37:51.599278 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:51.599181 4292 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7d43882-57a4-43f7-9ad9-c5174007f344-openshift-service-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:37:51.600682 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:51.600658 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7d43882-57a4-43f7-9ad9-c5174007f344-maas-api-tls" pod "f7d43882-57a4-43f7-9ad9-c5174007f344" (UID: "f7d43882-57a4-43f7-9ad9-c5174007f344"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:37:51.600812 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:51.600749 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7d43882-57a4-43f7-9ad9-c5174007f344-kube-api-access-bfb77" pod "f7d43882-57a4-43f7-9ad9-c5174007f344" (UID: "f7d43882-57a4-43f7-9ad9-c5174007f344"). InnerVolumeSpecName "kube-api-access-bfb77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:37:51.600812 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:51.600807 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7d43882-57a4-43f7-9ad9-c5174007f344-metrics-certs" pod "f7d43882-57a4-43f7-9ad9-c5174007f344" (UID: "f7d43882-57a4-43f7-9ad9-c5174007f344"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:37:51.700824 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:51.700737 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7d43882-57a4-43f7-9ad9-c5174007f344-maas-api-tls\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:37:51.700824 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:51.700766 4292 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7d43882-57a4-43f7-9ad9-c5174007f344-metrics-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:37:51.700824 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:51.700779 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfb77\" (UniqueName: \"kubernetes.io/projected/f7d43882-57a4-43f7-9ad9-c5174007f344-kube-api-access-bfb77\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:37:52.152737 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:52.152706 4292 generic.go:356] "Generic (PLEG): container finished" podID="f7d43882-57a4-43f7-9ad9-c5174007f344" containerID="b01f2cfbc155b30f4c01717cbdabc03df2223ce2e9b6a338ac525bbc06e43052" exitCode=0 Oct 30 20:37:52.152926 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:52.152757 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-98a19b-9477d9fdc-b2sls" Oct 30 20:37:52.152926 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:52.152794 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-98a19b-9477d9fdc-b2sls" event={"ID":"f7d43882-57a4-43f7-9ad9-c5174007f344","Type":"ContainerDied","Data":"b01f2cfbc155b30f4c01717cbdabc03df2223ce2e9b6a338ac525bbc06e43052"} Oct 30 20:37:52.152926 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:52.152841 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-98a19b-9477d9fdc-b2sls" event={"ID":"f7d43882-57a4-43f7-9ad9-c5174007f344","Type":"ContainerDied","Data":"86fd8b7688c4c9c08e762238036d1a43cf181b8343ff57a0dcb0ddc56387b50d"} Oct 30 20:37:52.152926 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:52.152864 4292 scope.go:122] "RemoveContainer" containerID="b01f2cfbc155b30f4c01717cbdabc03df2223ce2e9b6a338ac525bbc06e43052" Oct 30 20:37:52.162706 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:52.162503 4292 scope.go:122] "RemoveContainer" containerID="b01f2cfbc155b30f4c01717cbdabc03df2223ce2e9b6a338ac525bbc06e43052" Oct 30 20:37:52.162935 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:37:52.162764 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b01f2cfbc155b30f4c01717cbdabc03df2223ce2e9b6a338ac525bbc06e43052\": container with ID starting with b01f2cfbc155b30f4c01717cbdabc03df2223ce2e9b6a338ac525bbc06e43052 not found: ID does not exist" containerID="b01f2cfbc155b30f4c01717cbdabc03df2223ce2e9b6a338ac525bbc06e43052" Oct 30 20:37:52.162935 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:52.162784 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b01f2cfbc155b30f4c01717cbdabc03df2223ce2e9b6a338ac525bbc06e43052"} err="failed to get container status \"b01f2cfbc155b30f4c01717cbdabc03df2223ce2e9b6a338ac525bbc06e43052\": rpc error: code = NotFound desc = could not find container \"b01f2cfbc155b30f4c01717cbdabc03df2223ce2e9b6a338ac525bbc06e43052\": container with ID starting with b01f2cfbc155b30f4c01717cbdabc03df2223ce2e9b6a338ac525bbc06e43052 not found: ID does not exist" Oct 30 20:37:52.324756 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:52.324728 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-98a19b-9477d9fdc-b2sls"] Oct 30 20:37:52.375383 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:52.375337 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-98a19b-9477d9fdc-b2sls"] Oct 30 20:37:52.581607 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:52.581565 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7d43882-57a4-43f7-9ad9-c5174007f344" path="/var/lib/kubelet/pods/f7d43882-57a4-43f7-9ad9-c5174007f344/volumes" Oct 30 20:37:53.354921 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:53.354885 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-98a19b-h29wq"] Oct 30 20:37:53.405640 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:53.405613 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-98a19b-h29wq"] Oct 30 20:37:54.581179 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:54.581149 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3165f4d2-0fa3-4358-8204-d1e5b46e5d41" path="/var/lib/kubelet/pods/3165f4d2-0fa3-4358-8204-d1e5b46e5d41/volumes" Oct 30 20:37:54.686520 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:54.686486 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-547082-hfgnm"] Oct 30 20:37:54.737131 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:54.737101 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-547082-hfgnm"] Oct 30 20:37:56.580444 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:56.580408 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25776845-32c3-4c70-aa99-26fe1fa5ecd0" path="/var/lib/kubelet/pods/25776845-32c3-4c70-aa99-26fe1fa5ecd0/volumes" Oct 30 20:37:58.387692 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:37:58.387665 4292 scope.go:122] "RemoveContainer" containerID="b0c5e9641c82a20cdbdcb390e3be090df0efece3230889829254871c960dcefe" Oct 30 20:38:02.120655 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:02.120621 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj"] Oct 30 20:38:02.121077 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:02.120810 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj" podUID="f28646cd-7d71-45f8-9a70-407819e9b293" containerName="istio-proxy" containerID="cri-o://b48c5601c5ab92d1034c2bf947365222a8e19aabb725e9e795a72501bcc80815" gracePeriod=30 Oct 30 20:38:03.081930 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:03.081893 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e4864d-678tv"] Oct 30 20:38:03.082244 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:03.082233 4292 cpu_manager.go:414] "containerMap: removing container" podUID="f7d43882-57a4-43f7-9ad9-c5174007f344" containerName="maas-api" Oct 30 20:38:03.082293 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:03.082245 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7d43882-57a4-43f7-9ad9-c5174007f344" containerName="maas-api" Oct 30 20:38:03.082293 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:03.082259 4292 cpu_manager.go:414] "containerMap: removing container" podUID="9d5226bf-9ab0-496b-b2f5-f0d730554365" containerName="istio-proxy" Oct 30 20:38:03.082293 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:03.082264 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d5226bf-9ab0-496b-b2f5-f0d730554365" containerName="istio-proxy" Oct 30 20:38:03.082293 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:03.082278 4292 cpu_manager.go:414] "containerMap: removing container" podUID="6d8780e4-df03-426f-8981-f355e0430b39" containerName="istio-proxy" Oct 30 20:38:03.082293 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:03.082283 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d8780e4-df03-426f-8981-f355e0430b39" containerName="istio-proxy" Oct 30 20:38:03.082470 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:03.082303 4292 cpu_manager.go:414] "containerMap: removing container" podUID="85408ab0-d248-46df-a359-6a213c10fd58" containerName="maas-api" Oct 30 20:38:03.082470 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:03.082308 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85408ab0-d248-46df-a359-6a213c10fd58" containerName="maas-api" Oct 30 20:38:03.082470 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:03.082374 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="85408ab0-d248-46df-a359-6a213c10fd58" containerName="maas-api" Oct 30 20:38:03.082470 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:03.082385 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7d43882-57a4-43f7-9ad9-c5174007f344" containerName="maas-api" Oct 30 20:38:03.082470 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:03.082390 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d5226bf-9ab0-496b-b2f5-f0d730554365" containerName="istio-proxy" Oct 30 20:38:03.082470 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:03.082399 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d8780e4-df03-426f-8981-f355e0430b39" containerName="istio-proxy" Oct 30 20:38:03.086960 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:03.086942 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e4864d-678tv" Oct 30 20:38:03.099292 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:03.099272 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c751573d-35c8-4adc-b980-6e171acb2d59-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-e4864d-678tv\" (UID: \"c751573d-35c8-4adc-b980-6e171acb2d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e4864d-678tv" Oct 30 20:38:03.099404 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:03.099331 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q66gh\" (UniqueName: \"kubernetes.io/projected/c751573d-35c8-4adc-b980-6e171acb2d59-kube-api-access-q66gh\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-e4864d-678tv\" (UID: \"c751573d-35c8-4adc-b980-6e171acb2d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e4864d-678tv" Oct 30 20:38:03.099452 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:03.099414 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c751573d-35c8-4adc-b980-6e171acb2d59-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-e4864d-678tv\" (UID: \"c751573d-35c8-4adc-b980-6e171acb2d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e4864d-678tv" Oct 30 20:38:03.140540 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:03.140512 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e4864d-678tv"] Oct 30 20:38:03.188776 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:03.188750 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:38:03.188914 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:03.188795 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\"" Oct 30 20:38:03.199874 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:03.199855 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q66gh\" (UniqueName: \"kubernetes.io/projected/c751573d-35c8-4adc-b980-6e171acb2d59-kube-api-access-q66gh\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-e4864d-678tv\" (UID: \"c751573d-35c8-4adc-b980-6e171acb2d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e4864d-678tv" Oct 30 20:38:03.199982 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:03.199880 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c751573d-35c8-4adc-b980-6e171acb2d59-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-e4864d-678tv\" (UID: \"c751573d-35c8-4adc-b980-6e171acb2d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e4864d-678tv" Oct 30 20:38:03.199982 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:03.199930 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c751573d-35c8-4adc-b980-6e171acb2d59-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-e4864d-678tv\" (UID: \"c751573d-35c8-4adc-b980-6e171acb2d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e4864d-678tv" Oct 30 20:38:03.200662 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:03.200641 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c751573d-35c8-4adc-b980-6e171acb2d59-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-e4864d-678tv\" (UID: \"c751573d-35c8-4adc-b980-6e171acb2d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e4864d-678tv" Oct 30 20:38:03.202282 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:03.202261 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c751573d-35c8-4adc-b980-6e171acb2d59-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-e4864d-678tv\" (UID: \"c751573d-35c8-4adc-b980-6e171acb2d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e4864d-678tv" Oct 30 20:38:03.264402 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:03.264372 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q66gh\" (UniqueName: \"kubernetes.io/projected/c751573d-35c8-4adc-b980-6e171acb2d59-kube-api-access-q66gh\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-e4864d-678tv\" (UID: \"c751573d-35c8-4adc-b980-6e171acb2d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e4864d-678tv" Oct 30 20:38:03.399975 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:03.399899 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e4864d-678tv" Oct 30 20:38:03.778701 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:03.778674 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e4864d-678tv"] Oct 30 20:38:03.780939 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:38:03.780912 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc751573d_35c8_4adc_b980_6e171acb2d59.slice/crio-81432c36b7f61459f1331e11becadd1b6d83aae4e0187191e583b53ba4d4ed32 WatchSource:0}: Error finding container 81432c36b7f61459f1331e11becadd1b6d83aae4e0187191e583b53ba4d4ed32: Status 404 returned error can't find the container with id 81432c36b7f61459f1331e11becadd1b6d83aae4e0187191e583b53ba4d4ed32 Oct 30 20:38:04.218319 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:04.218288 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e4864d-678tv" event={"ID":"c751573d-35c8-4adc-b980-6e171acb2d59","Type":"ContainerStarted","Data":"b81fabca144aff9f0b28a914ff9d6208ce0f5fa15266e5dd6c15280ad9f90672"} Oct 30 20:38:04.218704 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:04.218334 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e4864d-678tv" event={"ID":"c751573d-35c8-4adc-b980-6e171acb2d59","Type":"ContainerStarted","Data":"81432c36b7f61459f1331e11becadd1b6d83aae4e0187191e583b53ba4d4ed32"} Oct 30 20:38:04.348016 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:04.347920 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e4864d-678tv" podStartSLOduration=1.347904629 podStartE2EDuration="1.347904629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:38:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:04.346289427 +0000 UTC m=+4387.864604885" watchObservedRunningTime="2026-10-30 20:38:04.347904629 +0000 UTC m=+4387.866220085" Oct 30 20:38:05.224884 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:05.224851 4292 generic.go:356] "Generic (PLEG): container finished" podID="c751573d-35c8-4adc-b980-6e171acb2d59" containerID="b81fabca144aff9f0b28a914ff9d6208ce0f5fa15266e5dd6c15280ad9f90672" exitCode=0 Oct 30 20:38:05.225298 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:05.224937 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e4864d-678tv" event={"ID":"c751573d-35c8-4adc-b980-6e171acb2d59","Type":"ContainerDied","Data":"b81fabca144aff9f0b28a914ff9d6208ce0f5fa15266e5dd6c15280ad9f90672"} Oct 30 20:38:06.361030 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:06.361008 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e4864d-678tv" Oct 30 20:38:06.423806 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:06.423780 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c751573d-35c8-4adc-b980-6e171acb2d59-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c751573d-35c8-4adc-b980-6e171acb2d59-maas-api-xks-ca\") pod \"c751573d-35c8-4adc-b980-6e171acb2d59\" (UID: \"c751573d-35c8-4adc-b980-6e171acb2d59\") " Oct 30 20:38:06.423985 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:06.423878 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c751573d-35c8-4adc-b980-6e171acb2d59-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c751573d-35c8-4adc-b980-6e171acb2d59-maas-api-openshift-ca\") pod \"c751573d-35c8-4adc-b980-6e171acb2d59\" (UID: \"c751573d-35c8-4adc-b980-6e171acb2d59\") " Oct 30 20:38:06.423985 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:06.423909 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c751573d-35c8-4adc-b980-6e171acb2d59-kube-api-access-q66gh\" (UniqueName: \"kubernetes.io/projected/c751573d-35c8-4adc-b980-6e171acb2d59-kube-api-access-q66gh\") pod \"c751573d-35c8-4adc-b980-6e171acb2d59\" (UID: \"c751573d-35c8-4adc-b980-6e171acb2d59\") " Oct 30 20:38:06.424207 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:06.424183 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c751573d-35c8-4adc-b980-6e171acb2d59-maas-api-openshift-ca" pod "c751573d-35c8-4adc-b980-6e171acb2d59" (UID: "c751573d-35c8-4adc-b980-6e171acb2d59"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:38:06.425889 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:06.425858 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c751573d-35c8-4adc-b980-6e171acb2d59-maas-api-xks-ca" pod "c751573d-35c8-4adc-b980-6e171acb2d59" (UID: "c751573d-35c8-4adc-b980-6e171acb2d59"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:38:06.425994 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:06.425917 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c751573d-35c8-4adc-b980-6e171acb2d59-kube-api-access-q66gh" pod "c751573d-35c8-4adc-b980-6e171acb2d59" (UID: "c751573d-35c8-4adc-b980-6e171acb2d59"). InnerVolumeSpecName "kube-api-access-q66gh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:38:06.524598 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:06.524568 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c751573d-35c8-4adc-b980-6e171acb2d59-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:38:06.524598 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:06.524597 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c751573d-35c8-4adc-b980-6e171acb2d59-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:38:06.524804 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:06.524612 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q66gh\" (UniqueName: \"kubernetes.io/projected/c751573d-35c8-4adc-b980-6e171acb2d59-kube-api-access-q66gh\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:38:07.236092 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:07.236050 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e4864d-678tv" event={"ID":"c751573d-35c8-4adc-b980-6e171acb2d59","Type":"ContainerDied","Data":"81432c36b7f61459f1331e11becadd1b6d83aae4e0187191e583b53ba4d4ed32"} Oct 30 20:38:07.236092 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:07.236082 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e4864d-678tv" Oct 30 20:38:07.236092 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:07.236097 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81432c36b7f61459f1331e11becadd1b6d83aae4e0187191e583b53ba4d4ed32" Oct 30 20:38:07.367844 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:07.367822 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj" Oct 30 20:38:07.432684 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:07.432616 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f28646cd-7d71-45f8-9a70-407819e9b293-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f28646cd-7d71-45f8-9a70-407819e9b293-istiod-ca-cert\") pod \"f28646cd-7d71-45f8-9a70-407819e9b293\" (UID: \"f28646cd-7d71-45f8-9a70-407819e9b293\") " Oct 30 20:38:07.432684 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:07.432648 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f28646cd-7d71-45f8-9a70-407819e9b293-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f28646cd-7d71-45f8-9a70-407819e9b293-istio-podinfo\") pod \"f28646cd-7d71-45f8-9a70-407819e9b293\" (UID: \"f28646cd-7d71-45f8-9a70-407819e9b293\") " Oct 30 20:38:07.432684 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:07.432677 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f28646cd-7d71-45f8-9a70-407819e9b293-istio-token\" (UniqueName: \"kubernetes.io/projected/f28646cd-7d71-45f8-9a70-407819e9b293-istio-token\") pod \"f28646cd-7d71-45f8-9a70-407819e9b293\" (UID: \"f28646cd-7d71-45f8-9a70-407819e9b293\") " Oct 30 20:38:07.432945 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:07.432701 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f28646cd-7d71-45f8-9a70-407819e9b293-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f28646cd-7d71-45f8-9a70-407819e9b293-istio-data\") pod \"f28646cd-7d71-45f8-9a70-407819e9b293\" (UID: \"f28646cd-7d71-45f8-9a70-407819e9b293\") " Oct 30 20:38:07.432945 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:07.432736 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f28646cd-7d71-45f8-9a70-407819e9b293-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28646cd-7d71-45f8-9a70-407819e9b293-workload-socket\") pod \"f28646cd-7d71-45f8-9a70-407819e9b293\" (UID: \"f28646cd-7d71-45f8-9a70-407819e9b293\") " Oct 30 20:38:07.432945 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:07.432760 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f28646cd-7d71-45f8-9a70-407819e9b293-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f28646cd-7d71-45f8-9a70-407819e9b293-istio-envoy\") pod \"f28646cd-7d71-45f8-9a70-407819e9b293\" (UID: \"f28646cd-7d71-45f8-9a70-407819e9b293\") " Oct 30 20:38:07.432945 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:07.432782 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f28646cd-7d71-45f8-9a70-407819e9b293-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f28646cd-7d71-45f8-9a70-407819e9b293-workload-certs\") pod \"f28646cd-7d71-45f8-9a70-407819e9b293\" (UID: \"f28646cd-7d71-45f8-9a70-407819e9b293\") " Oct 30 20:38:07.432945 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:07.432829 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f28646cd-7d71-45f8-9a70-407819e9b293-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28646cd-7d71-45f8-9a70-407819e9b293-credential-socket\") pod \"f28646cd-7d71-45f8-9a70-407819e9b293\" (UID: \"f28646cd-7d71-45f8-9a70-407819e9b293\") " Oct 30 20:38:07.432945 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:07.432857 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f28646cd-7d71-45f8-9a70-407819e9b293-kube-api-access-bcn2s\" (UniqueName: \"kubernetes.io/projected/f28646cd-7d71-45f8-9a70-407819e9b293-kube-api-access-bcn2s\") pod \"f28646cd-7d71-45f8-9a70-407819e9b293\" (UID: \"f28646cd-7d71-45f8-9a70-407819e9b293\") " Oct 30 20:38:07.433274 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:07.433031 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f28646cd-7d71-45f8-9a70-407819e9b293-workload-socket" pod "f28646cd-7d71-45f8-9a70-407819e9b293" (UID: "f28646cd-7d71-45f8-9a70-407819e9b293"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:38:07.433274 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:07.433054 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f28646cd-7d71-45f8-9a70-407819e9b293-istio-data" pod "f28646cd-7d71-45f8-9a70-407819e9b293" (UID: "f28646cd-7d71-45f8-9a70-407819e9b293"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:38:07.433274 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:07.433163 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f28646cd-7d71-45f8-9a70-407819e9b293-istio-data\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:38:07.433274 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:07.433179 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28646cd-7d71-45f8-9a70-407819e9b293-workload-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:38:07.433554 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:07.433518 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f28646cd-7d71-45f8-9a70-407819e9b293-istiod-ca-cert" pod "f28646cd-7d71-45f8-9a70-407819e9b293" (UID: "f28646cd-7d71-45f8-9a70-407819e9b293"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:38:07.433554 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:07.433541 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f28646cd-7d71-45f8-9a70-407819e9b293-workload-certs" pod "f28646cd-7d71-45f8-9a70-407819e9b293" (UID: "f28646cd-7d71-45f8-9a70-407819e9b293"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:38:07.433673 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:07.433605 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f28646cd-7d71-45f8-9a70-407819e9b293-credential-socket" pod "f28646cd-7d71-45f8-9a70-407819e9b293" (UID: "f28646cd-7d71-45f8-9a70-407819e9b293"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:38:07.435413 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:07.435387 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f28646cd-7d71-45f8-9a70-407819e9b293-kube-api-access-bcn2s" pod "f28646cd-7d71-45f8-9a70-407819e9b293" (UID: "f28646cd-7d71-45f8-9a70-407819e9b293"). InnerVolumeSpecName "kube-api-access-bcn2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:38:07.435517 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:07.435415 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f28646cd-7d71-45f8-9a70-407819e9b293-istio-podinfo" pod "f28646cd-7d71-45f8-9a70-407819e9b293" (UID: "f28646cd-7d71-45f8-9a70-407819e9b293"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:38:07.435517 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:07.435415 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f28646cd-7d71-45f8-9a70-407819e9b293-istio-envoy" pod "f28646cd-7d71-45f8-9a70-407819e9b293" (UID: "f28646cd-7d71-45f8-9a70-407819e9b293"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:38:07.435517 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:07.435419 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f28646cd-7d71-45f8-9a70-407819e9b293-istio-token" pod "f28646cd-7d71-45f8-9a70-407819e9b293" (UID: "f28646cd-7d71-45f8-9a70-407819e9b293"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:38:07.534488 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:07.534462 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f28646cd-7d71-45f8-9a70-407819e9b293-istiod-ca-cert\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:38:07.534488 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:07.534485 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f28646cd-7d71-45f8-9a70-407819e9b293-istio-podinfo\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:38:07.534684 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:07.534496 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f28646cd-7d71-45f8-9a70-407819e9b293-istio-token\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:38:07.534684 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:07.534505 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f28646cd-7d71-45f8-9a70-407819e9b293-istio-envoy\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:38:07.534684 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:07.534513 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f28646cd-7d71-45f8-9a70-407819e9b293-workload-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:38:07.534684 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:07.534521 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28646cd-7d71-45f8-9a70-407819e9b293-credential-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:38:07.534684 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:07.534529 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcn2s\" (UniqueName: \"kubernetes.io/projected/f28646cd-7d71-45f8-9a70-407819e9b293-kube-api-access-bcn2s\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:38:08.242918 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:08.242881 4292 generic.go:356] "Generic (PLEG): container finished" podID="f28646cd-7d71-45f8-9a70-407819e9b293" containerID="b48c5601c5ab92d1034c2bf947365222a8e19aabb725e9e795a72501bcc80815" exitCode=0 Oct 30 20:38:08.243093 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:08.242950 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj" Oct 30 20:38:08.243093 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:08.242961 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj" event={"ID":"f28646cd-7d71-45f8-9a70-407819e9b293","Type":"ContainerDied","Data":"b48c5601c5ab92d1034c2bf947365222a8e19aabb725e9e795a72501bcc80815"} Oct 30 20:38:08.243093 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:08.243038 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj" event={"ID":"f28646cd-7d71-45f8-9a70-407819e9b293","Type":"ContainerDied","Data":"4e5b059fd97399a0d6d85bda73dad99e5b487cb39ab0c1ac5b325bccab10e51e"} Oct 30 20:38:08.243093 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:08.243060 4292 scope.go:122] "RemoveContainer" containerID="b48c5601c5ab92d1034c2bf947365222a8e19aabb725e9e795a72501bcc80815" Oct 30 20:38:08.252166 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:08.252139 4292 scope.go:122] "RemoveContainer" containerID="814c4b7f8c40ba7416ba75b6addbf79d8f51ba597a7a28de3bdfe43d459c4208" Oct 30 20:38:08.260730 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:08.260713 4292 scope.go:122] "RemoveContainer" containerID="b48c5601c5ab92d1034c2bf947365222a8e19aabb725e9e795a72501bcc80815" Oct 30 20:38:08.260987 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:38:08.260967 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b48c5601c5ab92d1034c2bf947365222a8e19aabb725e9e795a72501bcc80815\": container with ID starting with b48c5601c5ab92d1034c2bf947365222a8e19aabb725e9e795a72501bcc80815 not found: ID does not exist" containerID="b48c5601c5ab92d1034c2bf947365222a8e19aabb725e9e795a72501bcc80815" Oct 30 20:38:08.261073 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:08.261002 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b48c5601c5ab92d1034c2bf947365222a8e19aabb725e9e795a72501bcc80815"} err="failed to get container status \"b48c5601c5ab92d1034c2bf947365222a8e19aabb725e9e795a72501bcc80815\": rpc error: code = NotFound desc = could not find container \"b48c5601c5ab92d1034c2bf947365222a8e19aabb725e9e795a72501bcc80815\": container with ID starting with b48c5601c5ab92d1034c2bf947365222a8e19aabb725e9e795a72501bcc80815 not found: ID does not exist" Oct 30 20:38:08.261073 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:08.261027 4292 scope.go:122] "RemoveContainer" containerID="814c4b7f8c40ba7416ba75b6addbf79d8f51ba597a7a28de3bdfe43d459c4208" Oct 30 20:38:08.261288 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:38:08.261271 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"814c4b7f8c40ba7416ba75b6addbf79d8f51ba597a7a28de3bdfe43d459c4208\": container with ID starting with 814c4b7f8c40ba7416ba75b6addbf79d8f51ba597a7a28de3bdfe43d459c4208 not found: ID does not exist" containerID="814c4b7f8c40ba7416ba75b6addbf79d8f51ba597a7a28de3bdfe43d459c4208" Oct 30 20:38:08.261335 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:08.261295 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"814c4b7f8c40ba7416ba75b6addbf79d8f51ba597a7a28de3bdfe43d459c4208"} err="failed to get container status \"814c4b7f8c40ba7416ba75b6addbf79d8f51ba597a7a28de3bdfe43d459c4208\": rpc error: code = NotFound desc = could not find container \"814c4b7f8c40ba7416ba75b6addbf79d8f51ba597a7a28de3bdfe43d459c4208\": container with ID starting with 814c4b7f8c40ba7416ba75b6addbf79d8f51ba597a7a28de3bdfe43d459c4208 not found: ID does not exist" Oct 30 20:38:08.463837 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:08.463784 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj"] Oct 30 20:38:08.514774 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:08.514693 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-547082-openshift-default-c85f94499-pq8lj"] Oct 30 20:38:08.580530 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:08.580498 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f28646cd-7d71-45f8-9a70-407819e9b293" path="/var/lib/kubelet/pods/f28646cd-7d71-45f8-9a70-407819e9b293/volumes" Oct 30 20:38:13.219550 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:38:13.219519 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:38:14.822906 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:14.822876 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e4864d-67698fc88c-t6ft7"] Oct 30 20:38:19.824019 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:19.823987 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e4864d-67698fc88c-t6ft7" podUID="cab76d4c-33c4-485e-b45d-368fb2137c63" containerName="payload-pre-processing" containerID="cri-o://ac24575d342cab09c7b1a16638c1e18f558d2ed88148c7c31eb642230844b747" gracePeriod=25 Oct 30 20:38:21.719555 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:21.719521 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e4864d-67698fc88c-t6ft7" podUID="cab76d4c-33c4-485e-b45d-368fb2137c63" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:38:31.620405 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:31.620371 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-273c1b/payload-processing-external-model-e2e-extmod-b-273c1b-5666q2xkc"] Oct 30 20:38:31.622662 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:31.620700 4292 cpu_manager.go:414] "containerMap: removing container" podUID="f28646cd-7d71-45f8-9a70-407819e9b293" containerName="istio-proxy" Oct 30 20:38:31.622662 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:31.620710 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f28646cd-7d71-45f8-9a70-407819e9b293" containerName="istio-proxy" Oct 30 20:38:31.622662 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:31.620736 4292 cpu_manager.go:414] "containerMap: removing container" podUID="c751573d-35c8-4adc-b980-6e171acb2d59" containerName="revoke-keys" Oct 30 20:38:31.622662 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:31.620741 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c751573d-35c8-4adc-b980-6e171acb2d59" containerName="revoke-keys" Oct 30 20:38:31.622662 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:31.620798 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="c751573d-35c8-4adc-b980-6e171acb2d59" containerName="revoke-keys" Oct 30 20:38:31.622662 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:31.620805 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="f28646cd-7d71-45f8-9a70-407819e9b293" containerName="istio-proxy" Oct 30 20:38:31.622662 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:31.620814 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="f28646cd-7d71-45f8-9a70-407819e9b293" containerName="istio-proxy" Oct 30 20:38:31.623557 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:31.623541 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-273c1b/payload-processing-external-model-e2e-extmod-b-273c1b-5666q2xkc" Oct 30 20:38:31.682063 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:31.682032 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-273c1b\"/\"routing-overlay\"" Oct 30 20:38:31.684216 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:31.684180 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-273c1b/payload-processing-external-model-e2e-extmod-b-273c1b-5666q2xkc"] Oct 30 20:38:31.720455 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:31.720416 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e4864d-67698fc88c-t6ft7" podUID="cab76d4c-33c4-485e-b45d-368fb2137c63" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:38:31.733122 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:31.733102 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-273c1b\"/\"agc-cred-857319a58a8b\"" Oct 30 20:38:31.733232 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:31.733211 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-273c1b\"/\"payload-processing-external-model-e2e--0ea6caa0-dockercfg-rrthh\"" Oct 30 20:38:31.754029 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:31.754005 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/94ce740c-20b4-4cd5-b060-898e41e38f62-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-273c1b-5666q2xkc\" (UID: \"94ce740c-20b4-4cd5-b060-898e41e38f62\") " pod="ai-tenant-e2e-extmod-b-273c1b/payload-processing-external-model-e2e-extmod-b-273c1b-5666q2xkc" Oct 30 20:38:31.754132 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:31.754053 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/94ce740c-20b4-4cd5-b060-898e41e38f62-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-273c1b-5666q2xkc\" (UID: \"94ce740c-20b4-4cd5-b060-898e41e38f62\") " pod="ai-tenant-e2e-extmod-b-273c1b/payload-processing-external-model-e2e-extmod-b-273c1b-5666q2xkc" Oct 30 20:38:31.754196 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:31.754180 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94ce740c-20b4-4cd5-b060-898e41e38f62-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-273c1b-5666q2xkc\" (UID: \"94ce740c-20b4-4cd5-b060-898e41e38f62\") " pod="ai-tenant-e2e-extmod-b-273c1b/payload-processing-external-model-e2e-extmod-b-273c1b-5666q2xkc" Oct 30 20:38:31.754260 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:31.754222 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zss5\" (UniqueName: \"kubernetes.io/projected/94ce740c-20b4-4cd5-b060-898e41e38f62-kube-api-access-9zss5\") pod \"payload-processing-external-model-e2e-extmod-b-273c1b-5666q2xkc\" (UID: \"94ce740c-20b4-4cd5-b060-898e41e38f62\") " pod="ai-tenant-e2e-extmod-b-273c1b/payload-processing-external-model-e2e-extmod-b-273c1b-5666q2xkc" Oct 30 20:38:31.782389 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:31.782339 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-273c1b\"/\"payload-processing-external-model-plugins-e2e-extmod-b-273c1b\"" Oct 30 20:38:31.782389 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:31.782378 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-273c1b\"/\"openshift-service-ca.crt\"" Oct 30 20:38:31.782524 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:31.782377 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-273c1b\"/\"kube-root-ca.crt\"" Oct 30 20:38:31.855105 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:31.855078 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/94ce740c-20b4-4cd5-b060-898e41e38f62-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-273c1b-5666q2xkc\" (UID: \"94ce740c-20b4-4cd5-b060-898e41e38f62\") " pod="ai-tenant-e2e-extmod-b-273c1b/payload-processing-external-model-e2e-extmod-b-273c1b-5666q2xkc" Oct 30 20:38:31.855225 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:31.855124 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94ce740c-20b4-4cd5-b060-898e41e38f62-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-273c1b-5666q2xkc\" (UID: \"94ce740c-20b4-4cd5-b060-898e41e38f62\") " pod="ai-tenant-e2e-extmod-b-273c1b/payload-processing-external-model-e2e-extmod-b-273c1b-5666q2xkc" Oct 30 20:38:31.855225 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:31.855144 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zss5\" (UniqueName: \"kubernetes.io/projected/94ce740c-20b4-4cd5-b060-898e41e38f62-kube-api-access-9zss5\") pod \"payload-processing-external-model-e2e-extmod-b-273c1b-5666q2xkc\" (UID: \"94ce740c-20b4-4cd5-b060-898e41e38f62\") " pod="ai-tenant-e2e-extmod-b-273c1b/payload-processing-external-model-e2e-extmod-b-273c1b-5666q2xkc" Oct 30 20:38:31.855225 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:31.855180 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/94ce740c-20b4-4cd5-b060-898e41e38f62-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-273c1b-5666q2xkc\" (UID: \"94ce740c-20b4-4cd5-b060-898e41e38f62\") " pod="ai-tenant-e2e-extmod-b-273c1b/payload-processing-external-model-e2e-extmod-b-273c1b-5666q2xkc" Oct 30 20:38:31.855790 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:31.855771 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/94ce740c-20b4-4cd5-b060-898e41e38f62-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-273c1b-5666q2xkc\" (UID: \"94ce740c-20b4-4cd5-b060-898e41e38f62\") " pod="ai-tenant-e2e-extmod-b-273c1b/payload-processing-external-model-e2e-extmod-b-273c1b-5666q2xkc" Oct 30 20:38:31.855857 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:31.855824 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94ce740c-20b4-4cd5-b060-898e41e38f62-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-273c1b-5666q2xkc\" (UID: \"94ce740c-20b4-4cd5-b060-898e41e38f62\") " pod="ai-tenant-e2e-extmod-b-273c1b/payload-processing-external-model-e2e-extmod-b-273c1b-5666q2xkc" Oct 30 20:38:31.857587 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:31.857567 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/94ce740c-20b4-4cd5-b060-898e41e38f62-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-273c1b-5666q2xkc\" (UID: \"94ce740c-20b4-4cd5-b060-898e41e38f62\") " pod="ai-tenant-e2e-extmod-b-273c1b/payload-processing-external-model-e2e-extmod-b-273c1b-5666q2xkc" Oct 30 20:38:31.918491 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:31.918428 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zss5\" (UniqueName: \"kubernetes.io/projected/94ce740c-20b4-4cd5-b060-898e41e38f62-kube-api-access-9zss5\") pod \"payload-processing-external-model-e2e-extmod-b-273c1b-5666q2xkc\" (UID: \"94ce740c-20b4-4cd5-b060-898e41e38f62\") " pod="ai-tenant-e2e-extmod-b-273c1b/payload-processing-external-model-e2e-extmod-b-273c1b-5666q2xkc" Oct 30 20:38:31.935400 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:31.935376 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-273c1b/payload-processing-external-model-e2e-extmod-b-273c1b-5666q2xkc" Oct 30 20:38:32.316880 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:32.316797 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-273c1b/payload-processing-external-model-e2e-extmod-b-273c1b-5666q2xkc"] Oct 30 20:38:32.320873 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:38:32.320844 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94ce740c_20b4_4cd5_b060_898e41e38f62.slice/crio-de6decd8ec7e56dc109bac8e8a9226e33ab5731d5f28d7054963f4012dd8df46 WatchSource:0}: Error finding container de6decd8ec7e56dc109bac8e8a9226e33ab5731d5f28d7054963f4012dd8df46: Status 404 returned error can't find the container with id de6decd8ec7e56dc109bac8e8a9226e33ab5731d5f28d7054963f4012dd8df46 Oct 30 20:38:32.363928 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:32.363903 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-273c1b/payload-processing-external-model-e2e-extmod-b-273c1b-5666q2xkc" event={"ID":"94ce740c-20b4-4cd5-b060-898e41e38f62","Type":"ContainerStarted","Data":"de6decd8ec7e56dc109bac8e8a9226e33ab5731d5f28d7054963f4012dd8df46"} Oct 30 20:38:33.370716 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:33.370683 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-273c1b/payload-processing-external-model-e2e-extmod-b-273c1b-5666q2xkc" event={"ID":"94ce740c-20b4-4cd5-b060-898e41e38f62","Type":"ContainerStarted","Data":"db00c143e2317049cc1732e5a718887cdfc7c1a7ec0e34fa3d4811cf5dcecfd6"} Oct 30 20:38:33.371107 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:33.370740 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-273c1b/payload-processing-external-model-e2e-extmod-b-273c1b-5666q2xkc" Oct 30 20:38:33.493250 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:33.493203 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-273c1b/payload-processing-external-model-e2e-extmod-b-273c1b-5666q2xkc" podStartSLOduration=2.493188414 podStartE2EDuration="2.493188414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:38:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:33.492369346 +0000 UTC m=+4417.010684794" watchObservedRunningTime="2026-10-30 20:38:33.493188414 +0000 UTC m=+4417.011503921" Oct 30 20:38:40.067646 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:40.067618 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e4864d-67698fc88c-t6ft7" Oct 30 20:38:40.228337 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:40.228244 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cab76d4c-33c4-485e-b45d-368fb2137c63-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cab76d4c-33c4-485e-b45d-368fb2137c63-plugins-config-volume\") pod \"cab76d4c-33c4-485e-b45d-368fb2137c63\" (UID: \"cab76d4c-33c4-485e-b45d-368fb2137c63\") " Oct 30 20:38:40.228337 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:40.228308 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cab76d4c-33c4-485e-b45d-368fb2137c63-kube-api-access-p2lgp\" (UniqueName: \"kubernetes.io/projected/cab76d4c-33c4-485e-b45d-368fb2137c63-kube-api-access-p2lgp\") pod \"cab76d4c-33c4-485e-b45d-368fb2137c63\" (UID: \"cab76d4c-33c4-485e-b45d-368fb2137c63\") " Oct 30 20:38:40.228699 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:40.228677 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cab76d4c-33c4-485e-b45d-368fb2137c63-plugins-config-volume" pod "cab76d4c-33c4-485e-b45d-368fb2137c63" (UID: "cab76d4c-33c4-485e-b45d-368fb2137c63"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:38:40.230536 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:40.230514 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cab76d4c-33c4-485e-b45d-368fb2137c63-kube-api-access-p2lgp" pod "cab76d4c-33c4-485e-b45d-368fb2137c63" (UID: "cab76d4c-33c4-485e-b45d-368fb2137c63"). InnerVolumeSpecName "kube-api-access-p2lgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:38:40.329717 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:40.329685 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cab76d4c-33c4-485e-b45d-368fb2137c63-plugins-config-volume\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:38:40.329717 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:40.329713 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2lgp\" (UniqueName: \"kubernetes.io/projected/cab76d4c-33c4-485e-b45d-368fb2137c63-kube-api-access-p2lgp\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:38:40.407684 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:40.407651 4292 generic.go:356] "Generic (PLEG): container finished" podID="cab76d4c-33c4-485e-b45d-368fb2137c63" containerID="ac24575d342cab09c7b1a16638c1e18f558d2ed88148c7c31eb642230844b747" exitCode=0 Oct 30 20:38:40.407867 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:40.407696 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e4864d-67698fc88c-t6ft7" Oct 30 20:38:40.407867 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:40.407735 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e4864d-67698fc88c-t6ft7" event={"ID":"cab76d4c-33c4-485e-b45d-368fb2137c63","Type":"ContainerDied","Data":"ac24575d342cab09c7b1a16638c1e18f558d2ed88148c7c31eb642230844b747"} Oct 30 20:38:40.407867 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:40.407792 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e4864d-67698fc88c-t6ft7" event={"ID":"cab76d4c-33c4-485e-b45d-368fb2137c63","Type":"ContainerDied","Data":"db9c4482c532c9f6b319f734610ef5b756cdc6410febfd11d9a6f68a4f73f703"} Oct 30 20:38:40.407867 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:40.407821 4292 scope.go:122] "RemoveContainer" containerID="ac24575d342cab09c7b1a16638c1e18f558d2ed88148c7c31eb642230844b747" Oct 30 20:38:40.416598 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:40.416582 4292 scope.go:122] "RemoveContainer" containerID="ac24575d342cab09c7b1a16638c1e18f558d2ed88148c7c31eb642230844b747" Oct 30 20:38:40.416859 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:38:40.416836 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac24575d342cab09c7b1a16638c1e18f558d2ed88148c7c31eb642230844b747\": container with ID starting with ac24575d342cab09c7b1a16638c1e18f558d2ed88148c7c31eb642230844b747 not found: ID does not exist" containerID="ac24575d342cab09c7b1a16638c1e18f558d2ed88148c7c31eb642230844b747" Oct 30 20:38:40.416933 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:40.416868 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac24575d342cab09c7b1a16638c1e18f558d2ed88148c7c31eb642230844b747"} err="failed to get container status \"ac24575d342cab09c7b1a16638c1e18f558d2ed88148c7c31eb642230844b747\": rpc error: code = NotFound desc = could not find container \"ac24575d342cab09c7b1a16638c1e18f558d2ed88148c7c31eb642230844b747\": container with ID starting with ac24575d342cab09c7b1a16638c1e18f558d2ed88148c7c31eb642230844b747 not found: ID does not exist" Oct 30 20:38:40.580642 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:40.580607 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e4864d-67698fc88c-t6ft7"] Oct 30 20:38:40.630162 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:40.630126 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e4864d-67698fc88c-t6ft7"] Oct 30 20:38:42.581068 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:42.581037 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cab76d4c-33c4-485e-b45d-368fb2137c63" path="/var/lib/kubelet/pods/cab76d4c-33c4-485e-b45d-368fb2137c63/volumes" Oct 30 20:38:43.218988 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:38:43.218944 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:38:44.378774 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:44.378742 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-273c1b/payload-processing-external-model-e2e-extmod-b-273c1b-5666q2xkc" Oct 30 20:38:53.399159 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:53.399124 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e4864d-678tv"] Oct 30 20:38:53.449655 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:53.449626 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e4864d-678tv"] Oct 30 20:38:54.580503 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:54.580470 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c751573d-35c8-4adc-b980-6e171acb2d59" path="/var/lib/kubelet/pods/c751573d-35c8-4adc-b980-6e171acb2d59/volumes" Oct 30 20:38:58.429629 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:58.429595 4292 scope.go:122] "RemoveContainer" containerID="538f721cfff19a1bac091250dffa8f0214b238565dbde21213de5d7ffe99d78e" Oct 30 20:38:58.440719 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:38:58.440663 4292 scope.go:122] "RemoveContainer" containerID="e981e78e721670733cd356f164a92a565190bd8d2ee9ad4c889db13cc4575784" Oct 30 20:39:01.636171 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:01.636140 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a74d15-vl7fc"] Oct 30 20:39:01.636592 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:01.636494 4292 cpu_manager.go:414] "containerMap: removing container" podUID="cab76d4c-33c4-485e-b45d-368fb2137c63" containerName="payload-pre-processing" Oct 30 20:39:01.636592 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:01.636508 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cab76d4c-33c4-485e-b45d-368fb2137c63" containerName="payload-pre-processing" Oct 30 20:39:01.636592 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:01.636547 4292 cpu_manager.go:414] "containerMap: removing container" podUID="f28646cd-7d71-45f8-9a70-407819e9b293" containerName="istio-proxy" Oct 30 20:39:01.636592 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:01.636553 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f28646cd-7d71-45f8-9a70-407819e9b293" containerName="istio-proxy" Oct 30 20:39:01.636731 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:01.636605 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="cab76d4c-33c4-485e-b45d-368fb2137c63" containerName="payload-pre-processing" Oct 30 20:39:01.641137 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:01.641118 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a74d15-vl7fc" Oct 30 20:39:01.695856 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:01.695829 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:39:01.695994 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:01.695872 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a74d15-vl7fc"] Oct 30 20:39:01.716835 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:01.716802 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsjxr\" (UniqueName: \"kubernetes.io/projected/4fe8fba7-ab2b-4b29-8c7c-9999ca750a97-kube-api-access-dsjxr\") pod \"maas-api-revoke-keys-e2e-worker-w0-a74d15-vl7fc\" (UID: \"4fe8fba7-ab2b-4b29-8c7c-9999ca750a97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a74d15-vl7fc" Oct 30 20:39:01.716954 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:01.716856 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fe8fba7-ab2b-4b29-8c7c-9999ca750a97-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-a74d15-vl7fc\" (UID: \"4fe8fba7-ab2b-4b29-8c7c-9999ca750a97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a74d15-vl7fc" Oct 30 20:39:01.716998 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:01.716950 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fe8fba7-ab2b-4b29-8c7c-9999ca750a97-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-a74d15-vl7fc\" (UID: \"4fe8fba7-ab2b-4b29-8c7c-9999ca750a97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a74d15-vl7fc" Oct 30 20:39:01.743904 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:01.743882 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\"" Oct 30 20:39:01.818361 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:01.818327 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsjxr\" (UniqueName: \"kubernetes.io/projected/4fe8fba7-ab2b-4b29-8c7c-9999ca750a97-kube-api-access-dsjxr\") pod \"maas-api-revoke-keys-e2e-worker-w0-a74d15-vl7fc\" (UID: \"4fe8fba7-ab2b-4b29-8c7c-9999ca750a97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a74d15-vl7fc" Oct 30 20:39:01.818521 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:01.818395 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fe8fba7-ab2b-4b29-8c7c-9999ca750a97-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-a74d15-vl7fc\" (UID: \"4fe8fba7-ab2b-4b29-8c7c-9999ca750a97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a74d15-vl7fc" Oct 30 20:39:01.818521 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:01.818429 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fe8fba7-ab2b-4b29-8c7c-9999ca750a97-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-a74d15-vl7fc\" (UID: \"4fe8fba7-ab2b-4b29-8c7c-9999ca750a97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a74d15-vl7fc" Oct 30 20:39:01.819043 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:01.819022 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fe8fba7-ab2b-4b29-8c7c-9999ca750a97-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-a74d15-vl7fc\" (UID: \"4fe8fba7-ab2b-4b29-8c7c-9999ca750a97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a74d15-vl7fc" Oct 30 20:39:01.820687 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:01.820667 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fe8fba7-ab2b-4b29-8c7c-9999ca750a97-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-a74d15-vl7fc\" (UID: \"4fe8fba7-ab2b-4b29-8c7c-9999ca750a97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a74d15-vl7fc" Oct 30 20:39:01.880863 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:01.880830 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsjxr\" (UniqueName: \"kubernetes.io/projected/4fe8fba7-ab2b-4b29-8c7c-9999ca750a97-kube-api-access-dsjxr\") pod \"maas-api-revoke-keys-e2e-worker-w0-a74d15-vl7fc\" (UID: \"4fe8fba7-ab2b-4b29-8c7c-9999ca750a97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a74d15-vl7fc" Oct 30 20:39:01.953415 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:01.953327 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a74d15-vl7fc" Oct 30 20:39:02.330841 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:02.330815 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a74d15-vl7fc"] Oct 30 20:39:02.332226 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:39:02.332201 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fe8fba7_ab2b_4b29_8c7c_9999ca750a97.slice/crio-ddcc917791f50eef5bee04137ddb7b6e3cdba6dcf7998c6dcc30ced26f3681c0 WatchSource:0}: Error finding container ddcc917791f50eef5bee04137ddb7b6e3cdba6dcf7998c6dcc30ced26f3681c0: Status 404 returned error can't find the container with id ddcc917791f50eef5bee04137ddb7b6e3cdba6dcf7998c6dcc30ced26f3681c0 Oct 30 20:39:02.519793 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:02.519761 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a74d15-vl7fc" event={"ID":"4fe8fba7-ab2b-4b29-8c7c-9999ca750a97","Type":"ContainerStarted","Data":"ddcc917791f50eef5bee04137ddb7b6e3cdba6dcf7998c6dcc30ced26f3681c0"} Oct 30 20:39:03.526956 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:03.526917 4292 generic.go:356] "Generic (PLEG): container finished" podID="4fe8fba7-ab2b-4b29-8c7c-9999ca750a97" containerID="28b052fe54e23972e2e6f410f7c8e24094089a196996e27e0e2e6918ea66779f" exitCode=0 Oct 30 20:39:03.527372 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:03.527009 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a74d15-vl7fc" event={"ID":"4fe8fba7-ab2b-4b29-8c7c-9999ca750a97","Type":"ContainerDied","Data":"28b052fe54e23972e2e6f410f7c8e24094089a196996e27e0e2e6918ea66779f"} Oct 30 20:39:04.666978 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:04.666954 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a74d15-vl7fc" Oct 30 20:39:04.746332 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:04.746302 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fe8fba7-ab2b-4b29-8c7c-9999ca750a97-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fe8fba7-ab2b-4b29-8c7c-9999ca750a97-maas-api-xks-ca\") pod \"4fe8fba7-ab2b-4b29-8c7c-9999ca750a97\" (UID: \"4fe8fba7-ab2b-4b29-8c7c-9999ca750a97\") " Oct 30 20:39:04.746505 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:04.746389 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fe8fba7-ab2b-4b29-8c7c-9999ca750a97-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fe8fba7-ab2b-4b29-8c7c-9999ca750a97-maas-api-openshift-ca\") pod \"4fe8fba7-ab2b-4b29-8c7c-9999ca750a97\" (UID: \"4fe8fba7-ab2b-4b29-8c7c-9999ca750a97\") " Oct 30 20:39:04.746505 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:04.746486 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fe8fba7-ab2b-4b29-8c7c-9999ca750a97-kube-api-access-dsjxr\" (UniqueName: \"kubernetes.io/projected/4fe8fba7-ab2b-4b29-8c7c-9999ca750a97-kube-api-access-dsjxr\") pod \"4fe8fba7-ab2b-4b29-8c7c-9999ca750a97\" (UID: \"4fe8fba7-ab2b-4b29-8c7c-9999ca750a97\") " Oct 30 20:39:04.746765 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:04.746734 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fe8fba7-ab2b-4b29-8c7c-9999ca750a97-maas-api-openshift-ca" pod "4fe8fba7-ab2b-4b29-8c7c-9999ca750a97" (UID: "4fe8fba7-ab2b-4b29-8c7c-9999ca750a97"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:39:04.748294 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:04.748273 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fe8fba7-ab2b-4b29-8c7c-9999ca750a97-maas-api-xks-ca" pod "4fe8fba7-ab2b-4b29-8c7c-9999ca750a97" (UID: "4fe8fba7-ab2b-4b29-8c7c-9999ca750a97"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:39:04.748471 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:04.748450 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fe8fba7-ab2b-4b29-8c7c-9999ca750a97-kube-api-access-dsjxr" pod "4fe8fba7-ab2b-4b29-8c7c-9999ca750a97" (UID: "4fe8fba7-ab2b-4b29-8c7c-9999ca750a97"). InnerVolumeSpecName "kube-api-access-dsjxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:39:04.847980 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:04.847895 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fe8fba7-ab2b-4b29-8c7c-9999ca750a97-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:39:04.847980 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:04.847924 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsjxr\" (UniqueName: \"kubernetes.io/projected/4fe8fba7-ab2b-4b29-8c7c-9999ca750a97-kube-api-access-dsjxr\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:39:04.847980 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:04.847937 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fe8fba7-ab2b-4b29-8c7c-9999ca750a97-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:39:05.538857 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:05.538826 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a74d15-vl7fc" event={"ID":"4fe8fba7-ab2b-4b29-8c7c-9999ca750a97","Type":"ContainerDied","Data":"ddcc917791f50eef5bee04137ddb7b6e3cdba6dcf7998c6dcc30ced26f3681c0"} Oct 30 20:39:05.539030 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:05.538865 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a74d15-vl7fc" Oct 30 20:39:05.539030 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:05.538872 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ddcc917791f50eef5bee04137ddb7b6e3cdba6dcf7998c6dcc30ced26f3681c0" Oct 30 20:39:06.024337 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:06.024305 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a74d15/distinct-2-w0-a74d15-kserve-846bdb9948-ht54r"] Oct 30 20:39:06.046239 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:06.046211 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a74d15/premium-w0-a74d15-kserve-864f45fdfb-949gm"] Oct 30 20:39:10.443896 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:10.443823 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-273c1b/payload-processing-external-model-e2e-extmod-b-273c1b-df7442nt9"] Oct 30 20:39:10.444237 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:10.444223 4292 cpu_manager.go:414] "containerMap: removing container" podUID="4fe8fba7-ab2b-4b29-8c7c-9999ca750a97" containerName="revoke-keys" Oct 30 20:39:10.444293 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:10.444237 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fe8fba7-ab2b-4b29-8c7c-9999ca750a97" containerName="revoke-keys" Oct 30 20:39:10.444328 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:10.444312 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fe8fba7-ab2b-4b29-8c7c-9999ca750a97" containerName="revoke-keys" Oct 30 20:39:10.447266 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:10.447251 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-273c1b/payload-processing-external-model-e2e-extmod-b-273c1b-df7442nt9" Oct 30 20:39:10.502166 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:10.502129 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/063e3d1f-f005-4df0-8c02-79778d593a3c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-273c1b-df7442nt9\" (UID: \"063e3d1f-f005-4df0-8c02-79778d593a3c\") " pod="ai-tenant-e2e-extmod-b-273c1b/payload-processing-external-model-e2e-extmod-b-273c1b-df7442nt9" Oct 30 20:39:10.502370 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:10.502264 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/063e3d1f-f005-4df0-8c02-79778d593a3c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-273c1b-df7442nt9\" (UID: \"063e3d1f-f005-4df0-8c02-79778d593a3c\") " pod="ai-tenant-e2e-extmod-b-273c1b/payload-processing-external-model-e2e-extmod-b-273c1b-df7442nt9" Oct 30 20:39:10.502370 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:10.502333 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjrnw\" (UniqueName: \"kubernetes.io/projected/063e3d1f-f005-4df0-8c02-79778d593a3c-kube-api-access-pjrnw\") pod \"payload-processing-external-model-e2e-extmod-b-273c1b-df7442nt9\" (UID: \"063e3d1f-f005-4df0-8c02-79778d593a3c\") " pod="ai-tenant-e2e-extmod-b-273c1b/payload-processing-external-model-e2e-extmod-b-273c1b-df7442nt9" Oct 30 20:39:10.502516 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:10.502417 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/063e3d1f-f005-4df0-8c02-79778d593a3c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-273c1b-df7442nt9\" (UID: \"063e3d1f-f005-4df0-8c02-79778d593a3c\") " pod="ai-tenant-e2e-extmod-b-273c1b/payload-processing-external-model-e2e-extmod-b-273c1b-df7442nt9" Oct 30 20:39:10.502937 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:10.502917 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-273c1b/payload-processing-external-model-e2e-extmod-b-273c1b-df7442nt9"] Oct 30 20:39:10.551240 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:10.551213 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-273c1b\"/\"agc-cred-81c84e9b3da3\"" Oct 30 20:39:10.603530 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:10.603499 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/063e3d1f-f005-4df0-8c02-79778d593a3c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-273c1b-df7442nt9\" (UID: \"063e3d1f-f005-4df0-8c02-79778d593a3c\") " pod="ai-tenant-e2e-extmod-b-273c1b/payload-processing-external-model-e2e-extmod-b-273c1b-df7442nt9" Oct 30 20:39:10.603685 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:10.603536 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjrnw\" (UniqueName: \"kubernetes.io/projected/063e3d1f-f005-4df0-8c02-79778d593a3c-kube-api-access-pjrnw\") pod \"payload-processing-external-model-e2e-extmod-b-273c1b-df7442nt9\" (UID: \"063e3d1f-f005-4df0-8c02-79778d593a3c\") " pod="ai-tenant-e2e-extmod-b-273c1b/payload-processing-external-model-e2e-extmod-b-273c1b-df7442nt9" Oct 30 20:39:10.603685 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:10.603557 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/063e3d1f-f005-4df0-8c02-79778d593a3c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-273c1b-df7442nt9\" (UID: \"063e3d1f-f005-4df0-8c02-79778d593a3c\") " pod="ai-tenant-e2e-extmod-b-273c1b/payload-processing-external-model-e2e-extmod-b-273c1b-df7442nt9" Oct 30 20:39:10.603685 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:10.603590 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/063e3d1f-f005-4df0-8c02-79778d593a3c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-273c1b-df7442nt9\" (UID: \"063e3d1f-f005-4df0-8c02-79778d593a3c\") " pod="ai-tenant-e2e-extmod-b-273c1b/payload-processing-external-model-e2e-extmod-b-273c1b-df7442nt9" Oct 30 20:39:10.604242 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:10.604218 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/063e3d1f-f005-4df0-8c02-79778d593a3c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-273c1b-df7442nt9\" (UID: \"063e3d1f-f005-4df0-8c02-79778d593a3c\") " pod="ai-tenant-e2e-extmod-b-273c1b/payload-processing-external-model-e2e-extmod-b-273c1b-df7442nt9" Oct 30 20:39:10.604242 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:10.604237 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/063e3d1f-f005-4df0-8c02-79778d593a3c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-273c1b-df7442nt9\" (UID: \"063e3d1f-f005-4df0-8c02-79778d593a3c\") " pod="ai-tenant-e2e-extmod-b-273c1b/payload-processing-external-model-e2e-extmod-b-273c1b-df7442nt9" Oct 30 20:39:10.606123 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:10.606102 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/063e3d1f-f005-4df0-8c02-79778d593a3c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-273c1b-df7442nt9\" (UID: \"063e3d1f-f005-4df0-8c02-79778d593a3c\") " pod="ai-tenant-e2e-extmod-b-273c1b/payload-processing-external-model-e2e-extmod-b-273c1b-df7442nt9" Oct 30 20:39:10.666279 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:10.666248 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjrnw\" (UniqueName: \"kubernetes.io/projected/063e3d1f-f005-4df0-8c02-79778d593a3c-kube-api-access-pjrnw\") pod \"payload-processing-external-model-e2e-extmod-b-273c1b-df7442nt9\" (UID: \"063e3d1f-f005-4df0-8c02-79778d593a3c\") " pod="ai-tenant-e2e-extmod-b-273c1b/payload-processing-external-model-e2e-extmod-b-273c1b-df7442nt9" Oct 30 20:39:10.759845 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:10.759772 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-273c1b/payload-processing-external-model-e2e-extmod-b-273c1b-df7442nt9" Oct 30 20:39:11.138798 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:11.138771 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-273c1b/payload-processing-external-model-e2e-extmod-b-273c1b-df7442nt9"] Oct 30 20:39:11.140117 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:39:11.140087 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod063e3d1f_f005_4df0_8c02_79778d593a3c.slice/crio-2a4450de825e591fcca627f205f56ee5d232e2fa5a709a39d74f77ced6d946c0 WatchSource:0}: Error finding container 2a4450de825e591fcca627f205f56ee5d232e2fa5a709a39d74f77ced6d946c0: Status 404 returned error can't find the container with id 2a4450de825e591fcca627f205f56ee5d232e2fa5a709a39d74f77ced6d946c0 Oct 30 20:39:11.570676 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:11.570642 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-273c1b/payload-processing-external-model-e2e-extmod-b-273c1b-df7442nt9" event={"ID":"063e3d1f-f005-4df0-8c02-79778d593a3c","Type":"ContainerStarted","Data":"b6664465387584c718312ac946b0dec0eb14d50c5622377e616f0bc563092e30"} Oct 30 20:39:11.571264 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:11.570700 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-273c1b/payload-processing-external-model-e2e-extmod-b-273c1b-df7442nt9" event={"ID":"063e3d1f-f005-4df0-8c02-79778d593a3c","Type":"ContainerStarted","Data":"2a4450de825e591fcca627f205f56ee5d232e2fa5a709a39d74f77ced6d946c0"} Oct 30 20:39:11.571264 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:11.570759 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-273c1b/payload-processing-external-model-e2e-extmod-b-273c1b-df7442nt9" Oct 30 20:39:11.691007 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:11.690958 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-273c1b/payload-processing-external-model-e2e-extmod-b-273c1b-df7442nt9" podStartSLOduration=1.690944514 podStartE2EDuration="1.690944514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:39:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:39:11.68948573 +0000 UTC m=+4455.207801209" watchObservedRunningTime="2026-10-30 20:39:11.690944514 +0000 UTC m=+4455.209259969" Oct 30 20:39:13.219529 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:39:13.219492 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:39:15.803283 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:15.803246 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-46dadd/payload-processing-external-model-e2e-extmod-a-46dadd-7d9cs546b"] Oct 30 20:39:15.807017 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:15.806996 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-46dadd/payload-processing-external-model-e2e-extmod-a-46dadd-7d9cs546b" Oct 30 20:39:15.860994 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:15.860965 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-46dadd\"/\"agc-cred-0daeff60e77f\"" Oct 30 20:39:15.862469 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:15.862447 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-46dadd/payload-processing-external-model-e2e-extmod-a-46dadd-7d9cs546b"] Oct 30 20:39:15.911116 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:15.911080 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-46dadd\"/\"payload-processing-external-model-plugins-e2e-extmod-a-46dadd\"" Oct 30 20:39:15.911116 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:15.911080 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-46dadd\"/\"payload-processing-external-model-e2e--3817cc28-dockercfg-xjl9b\"" Oct 30 20:39:15.911116 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:15.911087 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-46dadd\"/\"openshift-service-ca.crt\"" Oct 30 20:39:15.911766 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:15.911748 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-46dadd\"/\"kube-root-ca.crt\"" Oct 30 20:39:15.911766 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:15.911751 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-46dadd\"/\"routing-overlay\"" Oct 30 20:39:15.951736 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:15.951707 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c46b9ba8-0281-4ca6-95bd-e50a7c083f82-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-46dadd-7d9cs546b\" (UID: \"c46b9ba8-0281-4ca6-95bd-e50a7c083f82\") " pod="ai-tenant-e2e-extmod-a-46dadd/payload-processing-external-model-e2e-extmod-a-46dadd-7d9cs546b" Oct 30 20:39:15.951858 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:15.951747 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c46b9ba8-0281-4ca6-95bd-e50a7c083f82-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-46dadd-7d9cs546b\" (UID: \"c46b9ba8-0281-4ca6-95bd-e50a7c083f82\") " pod="ai-tenant-e2e-extmod-a-46dadd/payload-processing-external-model-e2e-extmod-a-46dadd-7d9cs546b" Oct 30 20:39:15.951858 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:15.951826 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c46b9ba8-0281-4ca6-95bd-e50a7c083f82-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-46dadd-7d9cs546b\" (UID: \"c46b9ba8-0281-4ca6-95bd-e50a7c083f82\") " pod="ai-tenant-e2e-extmod-a-46dadd/payload-processing-external-model-e2e-extmod-a-46dadd-7d9cs546b" Oct 30 20:39:15.951941 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:15.951907 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nrpz\" (UniqueName: \"kubernetes.io/projected/c46b9ba8-0281-4ca6-95bd-e50a7c083f82-kube-api-access-5nrpz\") pod \"payload-processing-external-model-e2e-extmod-a-46dadd-7d9cs546b\" (UID: \"c46b9ba8-0281-4ca6-95bd-e50a7c083f82\") " pod="ai-tenant-e2e-extmod-a-46dadd/payload-processing-external-model-e2e-extmod-a-46dadd-7d9cs546b" Oct 30 20:39:16.053439 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:16.053332 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nrpz\" (UniqueName: \"kubernetes.io/projected/c46b9ba8-0281-4ca6-95bd-e50a7c083f82-kube-api-access-5nrpz\") pod \"payload-processing-external-model-e2e-extmod-a-46dadd-7d9cs546b\" (UID: \"c46b9ba8-0281-4ca6-95bd-e50a7c083f82\") " pod="ai-tenant-e2e-extmod-a-46dadd/payload-processing-external-model-e2e-extmod-a-46dadd-7d9cs546b" Oct 30 20:39:16.053565 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:16.053435 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c46b9ba8-0281-4ca6-95bd-e50a7c083f82-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-46dadd-7d9cs546b\" (UID: \"c46b9ba8-0281-4ca6-95bd-e50a7c083f82\") " pod="ai-tenant-e2e-extmod-a-46dadd/payload-processing-external-model-e2e-extmod-a-46dadd-7d9cs546b" Oct 30 20:39:16.053565 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:16.053489 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c46b9ba8-0281-4ca6-95bd-e50a7c083f82-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-46dadd-7d9cs546b\" (UID: \"c46b9ba8-0281-4ca6-95bd-e50a7c083f82\") " pod="ai-tenant-e2e-extmod-a-46dadd/payload-processing-external-model-e2e-extmod-a-46dadd-7d9cs546b" Oct 30 20:39:16.053565 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:16.053541 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c46b9ba8-0281-4ca6-95bd-e50a7c083f82-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-46dadd-7d9cs546b\" (UID: \"c46b9ba8-0281-4ca6-95bd-e50a7c083f82\") " pod="ai-tenant-e2e-extmod-a-46dadd/payload-processing-external-model-e2e-extmod-a-46dadd-7d9cs546b" Oct 30 20:39:16.053684 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:39:16.053660 4292 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-46dadd/agc-cred-0daeff60e77f: secret "agc-cred-0daeff60e77f" not found Oct 30 20:39:16.053684 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:39:16.053676 4292 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-46dadd/payload-processing-external-model-e2e-extmod-a-46dadd-7d9cs546b: secret "agc-cred-0daeff60e77f" not found Oct 30 20:39:16.053750 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:39:16.053729 4292 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c46b9ba8-0281-4ca6-95bd-e50a7c083f82-provider-credentials podName:c46b9ba8-0281-4ca6-95bd-e50a7c083f82 nodeName:}" failed. No retries permitted until 2026-10-30 20:39:16.553709899 +0000 UTC m=+4460.072025351 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c46b9ba8-0281-4ca6-95bd-e50a7c083f82-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-46dadd-7d9cs546b" (UID: "c46b9ba8-0281-4ca6-95bd-e50a7c083f82") : secret "agc-cred-0daeff60e77f" not found Oct 30 20:39:16.054133 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:16.054112 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c46b9ba8-0281-4ca6-95bd-e50a7c083f82-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-46dadd-7d9cs546b\" (UID: \"c46b9ba8-0281-4ca6-95bd-e50a7c083f82\") " pod="ai-tenant-e2e-extmod-a-46dadd/payload-processing-external-model-e2e-extmod-a-46dadd-7d9cs546b" Oct 30 20:39:16.054239 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:16.054221 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c46b9ba8-0281-4ca6-95bd-e50a7c083f82-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-46dadd-7d9cs546b\" (UID: \"c46b9ba8-0281-4ca6-95bd-e50a7c083f82\") " pod="ai-tenant-e2e-extmod-a-46dadd/payload-processing-external-model-e2e-extmod-a-46dadd-7d9cs546b" Oct 30 20:39:16.117823 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:16.117796 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nrpz\" (UniqueName: \"kubernetes.io/projected/c46b9ba8-0281-4ca6-95bd-e50a7c083f82-kube-api-access-5nrpz\") pod \"payload-processing-external-model-e2e-extmod-a-46dadd-7d9cs546b\" (UID: \"c46b9ba8-0281-4ca6-95bd-e50a7c083f82\") " pod="ai-tenant-e2e-extmod-a-46dadd/payload-processing-external-model-e2e-extmod-a-46dadd-7d9cs546b" Oct 30 20:39:16.557812 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:16.557782 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c46b9ba8-0281-4ca6-95bd-e50a7c083f82-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-46dadd-7d9cs546b\" (UID: \"c46b9ba8-0281-4ca6-95bd-e50a7c083f82\") " pod="ai-tenant-e2e-extmod-a-46dadd/payload-processing-external-model-e2e-extmod-a-46dadd-7d9cs546b" Oct 30 20:39:16.557991 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:39:16.557918 4292 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-46dadd/agc-cred-0daeff60e77f: secret "agc-cred-0daeff60e77f" not found Oct 30 20:39:16.557991 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:39:16.557934 4292 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-46dadd/payload-processing-external-model-e2e-extmod-a-46dadd-7d9cs546b: secret "agc-cred-0daeff60e77f" not found Oct 30 20:39:16.557991 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:39:16.557983 4292 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c46b9ba8-0281-4ca6-95bd-e50a7c083f82-provider-credentials podName:c46b9ba8-0281-4ca6-95bd-e50a7c083f82 nodeName:}" failed. No retries permitted until 2026-10-30 20:39:17.557967898 +0000 UTC m=+4461.076283333 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c46b9ba8-0281-4ca6-95bd-e50a7c083f82-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-46dadd-7d9cs546b" (UID: "c46b9ba8-0281-4ca6-95bd-e50a7c083f82") : secret "agc-cred-0daeff60e77f" not found Oct 30 20:39:17.193326 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:17.193281 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-273c1b/payload-processing-external-model-e2e-extmod-b-273c1b-df7442nt9"] Oct 30 20:39:17.196309 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:17.196287 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-273c1b/payload-processing-external-model-e2e-extmod-b-273c1b-df7442nt9" Oct 30 20:39:17.567561 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:17.567528 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c46b9ba8-0281-4ca6-95bd-e50a7c083f82-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-46dadd-7d9cs546b\" (UID: \"c46b9ba8-0281-4ca6-95bd-e50a7c083f82\") " pod="ai-tenant-e2e-extmod-a-46dadd/payload-processing-external-model-e2e-extmod-a-46dadd-7d9cs546b" Oct 30 20:39:17.567732 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:39:17.567670 4292 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-46dadd/agc-cred-0daeff60e77f: secret "agc-cred-0daeff60e77f" not found Oct 30 20:39:17.567732 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:39:17.567686 4292 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-46dadd/payload-processing-external-model-e2e-extmod-a-46dadd-7d9cs546b: secret "agc-cred-0daeff60e77f" not found Oct 30 20:39:17.567813 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:39:17.567742 4292 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c46b9ba8-0281-4ca6-95bd-e50a7c083f82-provider-credentials podName:c46b9ba8-0281-4ca6-95bd-e50a7c083f82 nodeName:}" failed. No retries permitted until 2026-10-30 20:39:19.567728196 +0000 UTC m=+4463.086043635 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c46b9ba8-0281-4ca6-95bd-e50a7c083f82-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-46dadd-7d9cs546b" (UID: "c46b9ba8-0281-4ca6-95bd-e50a7c083f82") : secret "agc-cred-0daeff60e77f" not found Oct 30 20:39:19.585639 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:19.585601 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c46b9ba8-0281-4ca6-95bd-e50a7c083f82-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-46dadd-7d9cs546b\" (UID: \"c46b9ba8-0281-4ca6-95bd-e50a7c083f82\") " pod="ai-tenant-e2e-extmod-a-46dadd/payload-processing-external-model-e2e-extmod-a-46dadd-7d9cs546b" Oct 30 20:39:19.586053 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:39:19.585683 4292 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-46dadd/agc-cred-0daeff60e77f: secret "agc-cred-0daeff60e77f" not found Oct 30 20:39:19.586053 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:39:19.585699 4292 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-46dadd/payload-processing-external-model-e2e-extmod-a-46dadd-7d9cs546b: secret "agc-cred-0daeff60e77f" not found Oct 30 20:39:19.586053 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:39:19.585747 4292 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c46b9ba8-0281-4ca6-95bd-e50a7c083f82-provider-credentials podName:c46b9ba8-0281-4ca6-95bd-e50a7c083f82 nodeName:}" failed. No retries permitted until 2026-10-30 20:39:23.585733686 +0000 UTC m=+4467.104049120 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c46b9ba8-0281-4ca6-95bd-e50a7c083f82-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-46dadd-7d9cs546b" (UID: "c46b9ba8-0281-4ca6-95bd-e50a7c083f82") : secret "agc-cred-0daeff60e77f" not found Oct 30 20:39:21.043995 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.043934 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-a74d15/distinct-2-w0-a74d15-kserve-846bdb9948-ht54r" podUID="f70232fa-7106-4ec6-8c8e-277c4ce4c719" containerName="main" containerID="cri-o://1aee1e96ea8dda1200b0f2ac7be9f0d402a0dcfda93a0286dd14f06a84f3d3e3" gracePeriod=45 Oct 30 20:39:21.064508 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.064477 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-a74d15/premium-w0-a74d15-kserve-864f45fdfb-949gm" podUID="eb0ced80-4c4f-41dc-883c-b1a21e0f11f8" containerName="main" containerID="cri-o://ab98b68c6892c684789cd6797cbeba1c02c6a1bab2ba608fd1b674c6e6351810" gracePeriod=45 Oct 30 20:39:21.287641 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.287607 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-46dadd/payload-processing-external-model-e2e-extmod-a-46dadd-7d9cs546b"] Oct 30 20:39:21.287907 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:39:21.287886 4292 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-46dadd/payload-processing-external-model-e2e-extmod-a-46dadd-7d9cs546b" podUID="c46b9ba8-0281-4ca6-95bd-e50a7c083f82" Oct 30 20:39:21.303961 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.303908 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a74d15/distinct-2-w0-a74d15-kserve-846bdb9948-ht54r" Oct 30 20:39:21.331875 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.331850 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a74d15/premium-w0-a74d15-kserve-864f45fdfb-949gm" Oct 30 20:39:21.402182 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.402149 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f70232fa-7106-4ec6-8c8e-277c4ce4c719-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f70232fa-7106-4ec6-8c8e-277c4ce4c719-tmp-dir\") pod \"f70232fa-7106-4ec6-8c8e-277c4ce4c719\" (UID: \"f70232fa-7106-4ec6-8c8e-277c4ce4c719\") " Oct 30 20:39:21.402388 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.402208 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb0ced80-4c4f-41dc-883c-b1a21e0f11f8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb0ced80-4c4f-41dc-883c-b1a21e0f11f8-model-cache\") pod \"eb0ced80-4c4f-41dc-883c-b1a21e0f11f8\" (UID: \"eb0ced80-4c4f-41dc-883c-b1a21e0f11f8\") " Oct 30 20:39:21.402388 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.402249 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb0ced80-4c4f-41dc-883c-b1a21e0f11f8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb0ced80-4c4f-41dc-883c-b1a21e0f11f8-tmp-dir\") pod \"eb0ced80-4c4f-41dc-883c-b1a21e0f11f8\" (UID: \"eb0ced80-4c4f-41dc-883c-b1a21e0f11f8\") " Oct 30 20:39:21.402388 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.402292 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f70232fa-7106-4ec6-8c8e-277c4ce4c719-tls-certs\" (UniqueName: \"kubernetes.io/secret/f70232fa-7106-4ec6-8c8e-277c4ce4c719-tls-certs\") pod \"f70232fa-7106-4ec6-8c8e-277c4ce4c719\" (UID: \"f70232fa-7106-4ec6-8c8e-277c4ce4c719\") " Oct 30 20:39:21.402388 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.402344 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb0ced80-4c4f-41dc-883c-b1a21e0f11f8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb0ced80-4c4f-41dc-883c-b1a21e0f11f8-dshm\") pod \"eb0ced80-4c4f-41dc-883c-b1a21e0f11f8\" (UID: \"eb0ced80-4c4f-41dc-883c-b1a21e0f11f8\") " Oct 30 20:39:21.402676 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.402435 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f70232fa-7106-4ec6-8c8e-277c4ce4c719-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f70232fa-7106-4ec6-8c8e-277c4ce4c719-dshm\") pod \"f70232fa-7106-4ec6-8c8e-277c4ce4c719\" (UID: \"f70232fa-7106-4ec6-8c8e-277c4ce4c719\") " Oct 30 20:39:21.402676 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.402465 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f70232fa-7106-4ec6-8c8e-277c4ce4c719-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f70232fa-7106-4ec6-8c8e-277c4ce4c719-model-cache\") pod \"f70232fa-7106-4ec6-8c8e-277c4ce4c719\" (UID: \"f70232fa-7106-4ec6-8c8e-277c4ce4c719\") " Oct 30 20:39:21.402676 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.402454 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f70232fa-7106-4ec6-8c8e-277c4ce4c719-tmp-dir" pod "f70232fa-7106-4ec6-8c8e-277c4ce4c719" (UID: "f70232fa-7106-4ec6-8c8e-277c4ce4c719"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:39:21.402676 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.402518 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb0ced80-4c4f-41dc-883c-b1a21e0f11f8-kube-api-access-zv7f4\" (UniqueName: \"kubernetes.io/projected/eb0ced80-4c4f-41dc-883c-b1a21e0f11f8-kube-api-access-zv7f4\") pod \"eb0ced80-4c4f-41dc-883c-b1a21e0f11f8\" (UID: \"eb0ced80-4c4f-41dc-883c-b1a21e0f11f8\") " Oct 30 20:39:21.402676 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.402536 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb0ced80-4c4f-41dc-883c-b1a21e0f11f8-tmp-dir" pod "eb0ced80-4c4f-41dc-883c-b1a21e0f11f8" (UID: "eb0ced80-4c4f-41dc-883c-b1a21e0f11f8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:39:21.402676 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.402550 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb0ced80-4c4f-41dc-883c-b1a21e0f11f8-model-cache" pod "eb0ced80-4c4f-41dc-883c-b1a21e0f11f8" (UID: "eb0ced80-4c4f-41dc-883c-b1a21e0f11f8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:39:21.402676 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.402553 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f70232fa-7106-4ec6-8c8e-277c4ce4c719-home\" (UniqueName: \"kubernetes.io/empty-dir/f70232fa-7106-4ec6-8c8e-277c4ce4c719-home\") pod \"f70232fa-7106-4ec6-8c8e-277c4ce4c719\" (UID: \"f70232fa-7106-4ec6-8c8e-277c4ce4c719\") " Oct 30 20:39:21.402676 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.402634 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb0ced80-4c4f-41dc-883c-b1a21e0f11f8-tls-certs\" (UniqueName: \"kubernetes.io/secret/eb0ced80-4c4f-41dc-883c-b1a21e0f11f8-tls-certs\") pod \"eb0ced80-4c4f-41dc-883c-b1a21e0f11f8\" (UID: \"eb0ced80-4c4f-41dc-883c-b1a21e0f11f8\") " Oct 30 20:39:21.403122 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.402701 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb0ced80-4c4f-41dc-883c-b1a21e0f11f8-home\" (UniqueName: \"kubernetes.io/empty-dir/eb0ced80-4c4f-41dc-883c-b1a21e0f11f8-home\") pod \"eb0ced80-4c4f-41dc-883c-b1a21e0f11f8\" (UID: \"eb0ced80-4c4f-41dc-883c-b1a21e0f11f8\") " Oct 30 20:39:21.403122 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.402761 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f70232fa-7106-4ec6-8c8e-277c4ce4c719-home" pod "f70232fa-7106-4ec6-8c8e-277c4ce4c719" (UID: "f70232fa-7106-4ec6-8c8e-277c4ce4c719"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:39:21.403122 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.402773 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f70232fa-7106-4ec6-8c8e-277c4ce4c719-kube-api-access-r248s\" (UniqueName: \"kubernetes.io/projected/f70232fa-7106-4ec6-8c8e-277c4ce4c719-kube-api-access-r248s\") pod \"f70232fa-7106-4ec6-8c8e-277c4ce4c719\" (UID: \"f70232fa-7106-4ec6-8c8e-277c4ce4c719\") " Oct 30 20:39:21.403122 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.402796 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f70232fa-7106-4ec6-8c8e-277c4ce4c719-model-cache" pod "f70232fa-7106-4ec6-8c8e-277c4ce4c719" (UID: "f70232fa-7106-4ec6-8c8e-277c4ce4c719"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:39:21.403122 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.402981 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb0ced80-4c4f-41dc-883c-b1a21e0f11f8-home" pod "eb0ced80-4c4f-41dc-883c-b1a21e0f11f8" (UID: "eb0ced80-4c4f-41dc-883c-b1a21e0f11f8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:39:21.403397 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.403176 4292 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f70232fa-7106-4ec6-8c8e-277c4ce4c719-home\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:39:21.403397 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.403192 4292 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb0ced80-4c4f-41dc-883c-b1a21e0f11f8-home\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:39:21.403397 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.403206 4292 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f70232fa-7106-4ec6-8c8e-277c4ce4c719-tmp-dir\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:39:21.403397 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.403218 4292 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb0ced80-4c4f-41dc-883c-b1a21e0f11f8-model-cache\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:39:21.403397 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.403232 4292 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb0ced80-4c4f-41dc-883c-b1a21e0f11f8-tmp-dir\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:39:21.403397 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.403245 4292 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f70232fa-7106-4ec6-8c8e-277c4ce4c719-model-cache\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:39:21.405197 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.405161 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f70232fa-7106-4ec6-8c8e-277c4ce4c719-tls-certs" pod "f70232fa-7106-4ec6-8c8e-277c4ce4c719" (UID: "f70232fa-7106-4ec6-8c8e-277c4ce4c719"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:39:21.405309 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.405243 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb0ced80-4c4f-41dc-883c-b1a21e0f11f8-kube-api-access-zv7f4" pod "eb0ced80-4c4f-41dc-883c-b1a21e0f11f8" (UID: "eb0ced80-4c4f-41dc-883c-b1a21e0f11f8"). InnerVolumeSpecName "kube-api-access-zv7f4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:39:21.405309 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.405268 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f70232fa-7106-4ec6-8c8e-277c4ce4c719-dshm" pod "f70232fa-7106-4ec6-8c8e-277c4ce4c719" (UID: "f70232fa-7106-4ec6-8c8e-277c4ce4c719"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:39:21.405448 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.405303 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f70232fa-7106-4ec6-8c8e-277c4ce4c719-kube-api-access-r248s" pod "f70232fa-7106-4ec6-8c8e-277c4ce4c719" (UID: "f70232fa-7106-4ec6-8c8e-277c4ce4c719"). InnerVolumeSpecName "kube-api-access-r248s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:39:21.405448 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.405307 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb0ced80-4c4f-41dc-883c-b1a21e0f11f8-tls-certs" pod "eb0ced80-4c4f-41dc-883c-b1a21e0f11f8" (UID: "eb0ced80-4c4f-41dc-883c-b1a21e0f11f8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:39:21.405515 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.405494 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb0ced80-4c4f-41dc-883c-b1a21e0f11f8-dshm" pod "eb0ced80-4c4f-41dc-883c-b1a21e0f11f8" (UID: "eb0ced80-4c4f-41dc-883c-b1a21e0f11f8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:39:21.504413 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.504378 4292 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb0ced80-4c4f-41dc-883c-b1a21e0f11f8-dshm\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:39:21.504413 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.504414 4292 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f70232fa-7106-4ec6-8c8e-277c4ce4c719-dshm\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:39:21.504627 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.504428 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv7f4\" (UniqueName: \"kubernetes.io/projected/eb0ced80-4c4f-41dc-883c-b1a21e0f11f8-kube-api-access-zv7f4\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:39:21.504627 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.504444 4292 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb0ced80-4c4f-41dc-883c-b1a21e0f11f8-tls-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:39:21.504627 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.504458 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r248s\" (UniqueName: \"kubernetes.io/projected/f70232fa-7106-4ec6-8c8e-277c4ce4c719-kube-api-access-r248s\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:39:21.504627 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.504472 4292 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f70232fa-7106-4ec6-8c8e-277c4ce4c719-tls-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:39:21.631691 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.631607 4292 generic.go:356] "Generic (PLEG): container finished" podID="f70232fa-7106-4ec6-8c8e-277c4ce4c719" containerID="1aee1e96ea8dda1200b0f2ac7be9f0d402a0dcfda93a0286dd14f06a84f3d3e3" exitCode=0 Oct 30 20:39:21.631691 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.631667 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a74d15/distinct-2-w0-a74d15-kserve-846bdb9948-ht54r" Oct 30 20:39:21.631923 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.631686 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a74d15/distinct-2-w0-a74d15-kserve-846bdb9948-ht54r" event={"ID":"f70232fa-7106-4ec6-8c8e-277c4ce4c719","Type":"ContainerDied","Data":"1aee1e96ea8dda1200b0f2ac7be9f0d402a0dcfda93a0286dd14f06a84f3d3e3"} Oct 30 20:39:21.631923 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.631734 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a74d15/distinct-2-w0-a74d15-kserve-846bdb9948-ht54r" event={"ID":"f70232fa-7106-4ec6-8c8e-277c4ce4c719","Type":"ContainerDied","Data":"7164bae6efcca39366d2cea355c9e7f7183bddf90fbba1c0dc5ba31b1ca2e5af"} Oct 30 20:39:21.631923 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.631752 4292 scope.go:122] "RemoveContainer" containerID="1aee1e96ea8dda1200b0f2ac7be9f0d402a0dcfda93a0286dd14f06a84f3d3e3" Oct 30 20:39:21.633404 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.633344 4292 generic.go:356] "Generic (PLEG): container finished" podID="eb0ced80-4c4f-41dc-883c-b1a21e0f11f8" containerID="ab98b68c6892c684789cd6797cbeba1c02c6a1bab2ba608fd1b674c6e6351810" exitCode=0 Oct 30 20:39:21.633551 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.633399 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a74d15/premium-w0-a74d15-kserve-864f45fdfb-949gm" event={"ID":"eb0ced80-4c4f-41dc-883c-b1a21e0f11f8","Type":"ContainerDied","Data":"ab98b68c6892c684789cd6797cbeba1c02c6a1bab2ba608fd1b674c6e6351810"} Oct 30 20:39:21.633551 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.633436 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a74d15/premium-w0-a74d15-kserve-864f45fdfb-949gm" Oct 30 20:39:21.633551 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.633440 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a74d15/premium-w0-a74d15-kserve-864f45fdfb-949gm" event={"ID":"eb0ced80-4c4f-41dc-883c-b1a21e0f11f8","Type":"ContainerDied","Data":"405a23108c79c77532f530d636d8b7053ec1860e0aae282100f4027d06b8d217"} Oct 30 20:39:21.633740 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.633604 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-46dadd/payload-processing-external-model-e2e-extmod-a-46dadd-7d9cs546b" Oct 30 20:39:21.642532 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.642511 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-46dadd/payload-processing-external-model-e2e-extmod-a-46dadd-7d9cs546b" Oct 30 20:39:21.642899 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.642882 4292 scope.go:122] "RemoveContainer" containerID="1aee1e96ea8dda1200b0f2ac7be9f0d402a0dcfda93a0286dd14f06a84f3d3e3" Oct 30 20:39:21.643263 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:39:21.643242 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1aee1e96ea8dda1200b0f2ac7be9f0d402a0dcfda93a0286dd14f06a84f3d3e3\": container with ID starting with 1aee1e96ea8dda1200b0f2ac7be9f0d402a0dcfda93a0286dd14f06a84f3d3e3 not found: ID does not exist" containerID="1aee1e96ea8dda1200b0f2ac7be9f0d402a0dcfda93a0286dd14f06a84f3d3e3" Oct 30 20:39:21.643398 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.643272 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aee1e96ea8dda1200b0f2ac7be9f0d402a0dcfda93a0286dd14f06a84f3d3e3"} err="failed to get container status \"1aee1e96ea8dda1200b0f2ac7be9f0d402a0dcfda93a0286dd14f06a84f3d3e3\": rpc error: code = NotFound desc = could not find container \"1aee1e96ea8dda1200b0f2ac7be9f0d402a0dcfda93a0286dd14f06a84f3d3e3\": container with ID starting with 1aee1e96ea8dda1200b0f2ac7be9f0d402a0dcfda93a0286dd14f06a84f3d3e3 not found: ID does not exist" Oct 30 20:39:21.643398 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.643296 4292 scope.go:122] "RemoveContainer" containerID="ab98b68c6892c684789cd6797cbeba1c02c6a1bab2ba608fd1b674c6e6351810" Oct 30 20:39:21.651917 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.651901 4292 scope.go:122] "RemoveContainer" containerID="ab98b68c6892c684789cd6797cbeba1c02c6a1bab2ba608fd1b674c6e6351810" Oct 30 20:39:21.652167 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:39:21.652148 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab98b68c6892c684789cd6797cbeba1c02c6a1bab2ba608fd1b674c6e6351810\": container with ID starting with ab98b68c6892c684789cd6797cbeba1c02c6a1bab2ba608fd1b674c6e6351810 not found: ID does not exist" containerID="ab98b68c6892c684789cd6797cbeba1c02c6a1bab2ba608fd1b674c6e6351810" Oct 30 20:39:21.652216 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.652176 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab98b68c6892c684789cd6797cbeba1c02c6a1bab2ba608fd1b674c6e6351810"} err="failed to get container status \"ab98b68c6892c684789cd6797cbeba1c02c6a1bab2ba608fd1b674c6e6351810\": rpc error: code = NotFound desc = could not find container \"ab98b68c6892c684789cd6797cbeba1c02c6a1bab2ba608fd1b674c6e6351810\": container with ID starting with ab98b68c6892c684789cd6797cbeba1c02c6a1bab2ba608fd1b674c6e6351810 not found: ID does not exist" Oct 30 20:39:21.706300 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.706270 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c46b9ba8-0281-4ca6-95bd-e50a7c083f82-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c46b9ba8-0281-4ca6-95bd-e50a7c083f82-routing-overlay-volume\") pod \"c46b9ba8-0281-4ca6-95bd-e50a7c083f82\" (UID: \"c46b9ba8-0281-4ca6-95bd-e50a7c083f82\") " Oct 30 20:39:21.706441 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.706332 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c46b9ba8-0281-4ca6-95bd-e50a7c083f82-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c46b9ba8-0281-4ca6-95bd-e50a7c083f82-plugins-config-volume\") pod \"c46b9ba8-0281-4ca6-95bd-e50a7c083f82\" (UID: \"c46b9ba8-0281-4ca6-95bd-e50a7c083f82\") " Oct 30 20:39:21.706516 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.706484 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c46b9ba8-0281-4ca6-95bd-e50a7c083f82-kube-api-access-5nrpz\" (UniqueName: \"kubernetes.io/projected/c46b9ba8-0281-4ca6-95bd-e50a7c083f82-kube-api-access-5nrpz\") pod \"c46b9ba8-0281-4ca6-95bd-e50a7c083f82\" (UID: \"c46b9ba8-0281-4ca6-95bd-e50a7c083f82\") " Oct 30 20:39:21.706623 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.706604 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c46b9ba8-0281-4ca6-95bd-e50a7c083f82-routing-overlay-volume" pod "c46b9ba8-0281-4ca6-95bd-e50a7c083f82" (UID: "c46b9ba8-0281-4ca6-95bd-e50a7c083f82"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:39:21.706729 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.706705 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c46b9ba8-0281-4ca6-95bd-e50a7c083f82-plugins-config-volume" pod "c46b9ba8-0281-4ca6-95bd-e50a7c083f82" (UID: "c46b9ba8-0281-4ca6-95bd-e50a7c083f82"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:39:21.706838 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.706820 4292 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c46b9ba8-0281-4ca6-95bd-e50a7c083f82-routing-overlay-volume\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:39:21.706838 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.706837 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c46b9ba8-0281-4ca6-95bd-e50a7c083f82-plugins-config-volume\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:39:21.708439 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.708421 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c46b9ba8-0281-4ca6-95bd-e50a7c083f82-kube-api-access-5nrpz" pod "c46b9ba8-0281-4ca6-95bd-e50a7c083f82" (UID: "c46b9ba8-0281-4ca6-95bd-e50a7c083f82"). InnerVolumeSpecName "kube-api-access-5nrpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:39:21.800465 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.800426 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a74d15/premium-w0-a74d15-kserve-864f45fdfb-949gm"] Oct 30 20:39:21.804692 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.804664 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-a74d15/premium-w0-a74d15-kserve-864f45fdfb-949gm"] Oct 30 20:39:21.808154 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:21.808132 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nrpz\" (UniqueName: \"kubernetes.io/projected/c46b9ba8-0281-4ca6-95bd-e50a7c083f82-kube-api-access-5nrpz\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:39:22.035875 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:22.035696 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a74d15/distinct-2-w0-a74d15-kserve-846bdb9948-ht54r"] Oct 30 20:39:22.042140 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:22.042117 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-a74d15/distinct-2-w0-a74d15-kserve-846bdb9948-ht54r"] Oct 30 20:39:22.194184 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:22.194144 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-273c1b/payload-processing-external-model-e2e-extmod-b-273c1b-df7442nt9" podUID="063e3d1f-f005-4df0-8c02-79778d593a3c" containerName="payload-processing" containerID="cri-o://b6664465387584c718312ac946b0dec0eb14d50c5622377e616f0bc563092e30" gracePeriod=25 Oct 30 20:39:22.431210 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:22.431183 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-273c1b/payload-processing-external-model-e2e-extmod-b-273c1b-df7442nt9" Oct 30 20:39:22.514523 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:22.514498 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/063e3d1f-f005-4df0-8c02-79778d593a3c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/063e3d1f-f005-4df0-8c02-79778d593a3c-plugins-config-volume\") pod \"063e3d1f-f005-4df0-8c02-79778d593a3c\" (UID: \"063e3d1f-f005-4df0-8c02-79778d593a3c\") " Oct 30 20:39:22.514665 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:22.514570 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/063e3d1f-f005-4df0-8c02-79778d593a3c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/063e3d1f-f005-4df0-8c02-79778d593a3c-provider-credentials\") pod \"063e3d1f-f005-4df0-8c02-79778d593a3c\" (UID: \"063e3d1f-f005-4df0-8c02-79778d593a3c\") " Oct 30 20:39:22.514716 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:22.514673 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/063e3d1f-f005-4df0-8c02-79778d593a3c-kube-api-access-pjrnw\" (UniqueName: \"kubernetes.io/projected/063e3d1f-f005-4df0-8c02-79778d593a3c-kube-api-access-pjrnw\") pod \"063e3d1f-f005-4df0-8c02-79778d593a3c\" (UID: \"063e3d1f-f005-4df0-8c02-79778d593a3c\") " Oct 30 20:39:22.514749 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:22.514717 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/063e3d1f-f005-4df0-8c02-79778d593a3c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/063e3d1f-f005-4df0-8c02-79778d593a3c-routing-overlay-volume\") pod \"063e3d1f-f005-4df0-8c02-79778d593a3c\" (UID: \"063e3d1f-f005-4df0-8c02-79778d593a3c\") " Oct 30 20:39:22.514907 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:22.514885 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/063e3d1f-f005-4df0-8c02-79778d593a3c-plugins-config-volume" pod "063e3d1f-f005-4df0-8c02-79778d593a3c" (UID: "063e3d1f-f005-4df0-8c02-79778d593a3c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:39:22.515139 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:22.515120 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/063e3d1f-f005-4df0-8c02-79778d593a3c-routing-overlay-volume" pod "063e3d1f-f005-4df0-8c02-79778d593a3c" (UID: "063e3d1f-f005-4df0-8c02-79778d593a3c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:39:22.515198 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:22.515142 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/063e3d1f-f005-4df0-8c02-79778d593a3c-plugins-config-volume\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:39:22.516752 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:22.516723 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/063e3d1f-f005-4df0-8c02-79778d593a3c-provider-credentials" pod "063e3d1f-f005-4df0-8c02-79778d593a3c" (UID: "063e3d1f-f005-4df0-8c02-79778d593a3c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:39:22.516860 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:22.516802 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/063e3d1f-f005-4df0-8c02-79778d593a3c-kube-api-access-pjrnw" pod "063e3d1f-f005-4df0-8c02-79778d593a3c" (UID: "063e3d1f-f005-4df0-8c02-79778d593a3c"). InnerVolumeSpecName "kube-api-access-pjrnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:39:22.581046 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:22.580973 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb0ced80-4c4f-41dc-883c-b1a21e0f11f8" path="/var/lib/kubelet/pods/eb0ced80-4c4f-41dc-883c-b1a21e0f11f8/volumes" Oct 30 20:39:22.581438 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:22.581424 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f70232fa-7106-4ec6-8c8e-277c4ce4c719" path="/var/lib/kubelet/pods/f70232fa-7106-4ec6-8c8e-277c4ce4c719/volumes" Oct 30 20:39:22.616157 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:22.616126 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjrnw\" (UniqueName: \"kubernetes.io/projected/063e3d1f-f005-4df0-8c02-79778d593a3c-kube-api-access-pjrnw\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:39:22.616157 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:22.616151 4292 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/063e3d1f-f005-4df0-8c02-79778d593a3c-routing-overlay-volume\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:39:22.616321 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:22.616167 4292 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/063e3d1f-f005-4df0-8c02-79778d593a3c-provider-credentials\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:39:22.641941 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:22.641913 4292 generic.go:356] "Generic (PLEG): container finished" podID="063e3d1f-f005-4df0-8c02-79778d593a3c" containerID="b6664465387584c718312ac946b0dec0eb14d50c5622377e616f0bc563092e30" exitCode=0 Oct 30 20:39:22.642083 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:22.641955 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-273c1b/payload-processing-external-model-e2e-extmod-b-273c1b-df7442nt9" Oct 30 20:39:22.642083 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:22.641978 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-46dadd/payload-processing-external-model-e2e-extmod-a-46dadd-7d9cs546b" Oct 30 20:39:22.642083 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:22.642002 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-273c1b/payload-processing-external-model-e2e-extmod-b-273c1b-df7442nt9" event={"ID":"063e3d1f-f005-4df0-8c02-79778d593a3c","Type":"ContainerDied","Data":"b6664465387584c718312ac946b0dec0eb14d50c5622377e616f0bc563092e30"} Oct 30 20:39:22.642083 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:22.642061 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-273c1b/payload-processing-external-model-e2e-extmod-b-273c1b-df7442nt9" event={"ID":"063e3d1f-f005-4df0-8c02-79778d593a3c","Type":"ContainerDied","Data":"2a4450de825e591fcca627f205f56ee5d232e2fa5a709a39d74f77ced6d946c0"} Oct 30 20:39:22.642313 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:22.642092 4292 scope.go:122] "RemoveContainer" containerID="b6664465387584c718312ac946b0dec0eb14d50c5622377e616f0bc563092e30" Oct 30 20:39:22.650341 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:22.650316 4292 scope.go:122] "RemoveContainer" containerID="b6664465387584c718312ac946b0dec0eb14d50c5622377e616f0bc563092e30" Oct 30 20:39:22.650635 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:39:22.650615 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6664465387584c718312ac946b0dec0eb14d50c5622377e616f0bc563092e30\": container with ID starting with b6664465387584c718312ac946b0dec0eb14d50c5622377e616f0bc563092e30 not found: ID does not exist" containerID="b6664465387584c718312ac946b0dec0eb14d50c5622377e616f0bc563092e30" Oct 30 20:39:22.650702 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:22.650643 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6664465387584c718312ac946b0dec0eb14d50c5622377e616f0bc563092e30"} err="failed to get container status \"b6664465387584c718312ac946b0dec0eb14d50c5622377e616f0bc563092e30\": rpc error: code = NotFound desc = could not find container \"b6664465387584c718312ac946b0dec0eb14d50c5622377e616f0bc563092e30\": container with ID starting with b6664465387584c718312ac946b0dec0eb14d50c5622377e616f0bc563092e30 not found: ID does not exist" Oct 30 20:39:22.888268 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:22.888191 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-46dadd/payload-processing-external-model-e2e-extmod-a-46dadd-7d9cs546b"] Oct 30 20:39:22.892892 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:22.892864 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-46dadd/payload-processing-external-model-e2e-extmod-a-46dadd-7d9cs546b"] Oct 30 20:39:23.005000 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:23.004968 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-273c1b/payload-processing-external-model-e2e-extmod-b-273c1b-df7442nt9"] Oct 30 20:39:23.009641 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:23.009612 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-273c1b/payload-processing-external-model-e2e-extmod-b-273c1b-df7442nt9"] Oct 30 20:39:23.020031 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:23.020010 4292 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c46b9ba8-0281-4ca6-95bd-e50a7c083f82-provider-credentials\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:39:23.901929 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:23.901899 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-997c20ab-h8thk"] Oct 30 20:39:23.902315 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:23.902220 4292 cpu_manager.go:414] "containerMap: removing container" podUID="063e3d1f-f005-4df0-8c02-79778d593a3c" containerName="payload-processing" Oct 30 20:39:23.902315 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:23.902230 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="063e3d1f-f005-4df0-8c02-79778d593a3c" containerName="payload-processing" Oct 30 20:39:23.902315 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:23.902242 4292 cpu_manager.go:414] "containerMap: removing container" podUID="f70232fa-7106-4ec6-8c8e-277c4ce4c719" containerName="main" Oct 30 20:39:23.902315 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:23.902247 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f70232fa-7106-4ec6-8c8e-277c4ce4c719" containerName="main" Oct 30 20:39:23.902315 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:23.902265 4292 cpu_manager.go:414] "containerMap: removing container" podUID="eb0ced80-4c4f-41dc-883c-b1a21e0f11f8" containerName="main" Oct 30 20:39:23.902315 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:23.902270 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb0ced80-4c4f-41dc-883c-b1a21e0f11f8" containerName="main" Oct 30 20:39:23.902523 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:23.902325 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb0ced80-4c4f-41dc-883c-b1a21e0f11f8" containerName="main" Oct 30 20:39:23.902523 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:23.902336 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="063e3d1f-f005-4df0-8c02-79778d593a3c" containerName="payload-processing" Oct 30 20:39:23.902523 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:23.902343 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="f70232fa-7106-4ec6-8c8e-277c4ce4c719" containerName="main" Oct 30 20:39:23.906704 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:23.906687 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-997c20ab-h8thk" Oct 30 20:39:23.961401 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:23.961341 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b75042a-jbssd"] Oct 30 20:39:23.964893 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:23.964873 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-997c20ab-h8thk"] Oct 30 20:39:23.965020 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:23.964958 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b75042a-jbssd" Oct 30 20:39:24.009944 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:24.009911 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:39:24.010089 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:24.009960 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\"" Oct 30 20:39:24.019265 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:24.019241 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b75042a-jbssd"] Oct 30 20:39:24.029460 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:24.029434 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/160c9418-9850-4c39-8691-55202031b0a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-997c20ab-h8thk\" (UID: \"160c9418-9850-4c39-8691-55202031b0a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-997c20ab-h8thk" Oct 30 20:39:24.029592 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:24.029479 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/160c9418-9850-4c39-8691-55202031b0a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-997c20ab-h8thk\" (UID: \"160c9418-9850-4c39-8691-55202031b0a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-997c20ab-h8thk" Oct 30 20:39:24.029592 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:24.029503 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdqql\" (UniqueName: \"kubernetes.io/projected/160c9418-9850-4c39-8691-55202031b0a4-kube-api-access-tdqql\") pod \"maas-api-revoke-sub-997c20ab-h8thk\" (UID: \"160c9418-9850-4c39-8691-55202031b0a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-997c20ab-h8thk" Oct 30 20:39:24.130634 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:24.130600 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30ae8747-6c31-4160-8548-1a80e5305294-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b75042a-jbssd\" (UID: \"30ae8747-6c31-4160-8548-1a80e5305294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b75042a-jbssd" Oct 30 20:39:24.130826 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:24.130665 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/160c9418-9850-4c39-8691-55202031b0a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-997c20ab-h8thk\" (UID: \"160c9418-9850-4c39-8691-55202031b0a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-997c20ab-h8thk" Oct 30 20:39:24.130826 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:24.130737 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/160c9418-9850-4c39-8691-55202031b0a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-997c20ab-h8thk\" (UID: \"160c9418-9850-4c39-8691-55202031b0a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-997c20ab-h8thk" Oct 30 20:39:24.130826 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:24.130774 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdqql\" (UniqueName: \"kubernetes.io/projected/160c9418-9850-4c39-8691-55202031b0a4-kube-api-access-tdqql\") pod \"maas-api-revoke-sub-997c20ab-h8thk\" (UID: \"160c9418-9850-4c39-8691-55202031b0a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-997c20ab-h8thk" Oct 30 20:39:24.130826 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:24.130794 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nslq\" (UniqueName: \"kubernetes.io/projected/30ae8747-6c31-4160-8548-1a80e5305294-kube-api-access-4nslq\") pod \"maas-api-revoke-sub-6b75042a-jbssd\" (UID: \"30ae8747-6c31-4160-8548-1a80e5305294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b75042a-jbssd" Oct 30 20:39:24.130826 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:24.130815 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30ae8747-6c31-4160-8548-1a80e5305294-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b75042a-jbssd\" (UID: \"30ae8747-6c31-4160-8548-1a80e5305294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b75042a-jbssd" Oct 30 20:39:24.131504 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:24.131484 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/160c9418-9850-4c39-8691-55202031b0a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-997c20ab-h8thk\" (UID: \"160c9418-9850-4c39-8691-55202031b0a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-997c20ab-h8thk" Oct 30 20:39:24.133170 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:24.133150 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/160c9418-9850-4c39-8691-55202031b0a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-997c20ab-h8thk\" (UID: \"160c9418-9850-4c39-8691-55202031b0a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-997c20ab-h8thk" Oct 30 20:39:24.197229 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:24.197163 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdqql\" (UniqueName: \"kubernetes.io/projected/160c9418-9850-4c39-8691-55202031b0a4-kube-api-access-tdqql\") pod \"maas-api-revoke-sub-997c20ab-h8thk\" (UID: \"160c9418-9850-4c39-8691-55202031b0a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-997c20ab-h8thk" Oct 30 20:39:24.219004 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:24.218958 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-997c20ab-h8thk" Oct 30 20:39:24.231459 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:24.231434 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nslq\" (UniqueName: \"kubernetes.io/projected/30ae8747-6c31-4160-8548-1a80e5305294-kube-api-access-4nslq\") pod \"maas-api-revoke-sub-6b75042a-jbssd\" (UID: \"30ae8747-6c31-4160-8548-1a80e5305294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b75042a-jbssd" Oct 30 20:39:24.231574 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:24.231477 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30ae8747-6c31-4160-8548-1a80e5305294-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b75042a-jbssd\" (UID: \"30ae8747-6c31-4160-8548-1a80e5305294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b75042a-jbssd" Oct 30 20:39:24.231574 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:24.231542 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30ae8747-6c31-4160-8548-1a80e5305294-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b75042a-jbssd\" (UID: \"30ae8747-6c31-4160-8548-1a80e5305294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b75042a-jbssd" Oct 30 20:39:24.232270 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:24.232246 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30ae8747-6c31-4160-8548-1a80e5305294-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b75042a-jbssd\" (UID: \"30ae8747-6c31-4160-8548-1a80e5305294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b75042a-jbssd" Oct 30 20:39:24.233921 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:24.233898 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30ae8747-6c31-4160-8548-1a80e5305294-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b75042a-jbssd\" (UID: \"30ae8747-6c31-4160-8548-1a80e5305294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b75042a-jbssd" Oct 30 20:39:24.295007 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:24.294964 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nslq\" (UniqueName: \"kubernetes.io/projected/30ae8747-6c31-4160-8548-1a80e5305294-kube-api-access-4nslq\") pod \"maas-api-revoke-sub-6b75042a-jbssd\" (UID: \"30ae8747-6c31-4160-8548-1a80e5305294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b75042a-jbssd" Oct 30 20:39:24.449329 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:24.449238 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1301e352-zpl5b"] Oct 30 20:39:24.453869 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:24.453847 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1301e352-zpl5b" Oct 30 20:39:24.508843 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:24.508810 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1301e352-zpl5b"] Oct 30 20:39:24.534530 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:24.534499 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54spr\" (UniqueName: \"kubernetes.io/projected/9ac38922-9d8d-4005-b611-1f129ccc4f69-kube-api-access-54spr\") pod \"maas-api-revoke-sub-1301e352-zpl5b\" (UID: \"9ac38922-9d8d-4005-b611-1f129ccc4f69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1301e352-zpl5b" Oct 30 20:39:24.534689 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:24.534535 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ac38922-9d8d-4005-b611-1f129ccc4f69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1301e352-zpl5b\" (UID: \"9ac38922-9d8d-4005-b611-1f129ccc4f69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1301e352-zpl5b" Oct 30 20:39:24.534689 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:24.534644 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ac38922-9d8d-4005-b611-1f129ccc4f69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1301e352-zpl5b\" (UID: \"9ac38922-9d8d-4005-b611-1f129ccc4f69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1301e352-zpl5b" Oct 30 20:39:24.577112 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:24.577085 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b75042a-jbssd" Oct 30 20:39:24.580677 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:24.580646 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="063e3d1f-f005-4df0-8c02-79778d593a3c" path="/var/lib/kubelet/pods/063e3d1f-f005-4df0-8c02-79778d593a3c/volumes" Oct 30 20:39:24.581016 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:24.581001 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c46b9ba8-0281-4ca6-95bd-e50a7c083f82" path="/var/lib/kubelet/pods/c46b9ba8-0281-4ca6-95bd-e50a7c083f82/volumes" Oct 30 20:39:24.595769 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:24.595748 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-997c20ab-h8thk"] Oct 30 20:39:24.597665 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:39:24.597646 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod160c9418_9850_4c39_8691_55202031b0a4.slice/crio-d964b4e29f1b38b257693ce7d7c85f7e549584b99d95c19686eaaaecb6bce758 WatchSource:0}: Error finding container d964b4e29f1b38b257693ce7d7c85f7e549584b99d95c19686eaaaecb6bce758: Status 404 returned error can't find the container with id d964b4e29f1b38b257693ce7d7c85f7e549584b99d95c19686eaaaecb6bce758 Oct 30 20:39:24.636104 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:24.636079 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ac38922-9d8d-4005-b611-1f129ccc4f69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1301e352-zpl5b\" (UID: \"9ac38922-9d8d-4005-b611-1f129ccc4f69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1301e352-zpl5b" Oct 30 20:39:24.636229 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:24.636172 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54spr\" (UniqueName: \"kubernetes.io/projected/9ac38922-9d8d-4005-b611-1f129ccc4f69-kube-api-access-54spr\") pod \"maas-api-revoke-sub-1301e352-zpl5b\" (UID: \"9ac38922-9d8d-4005-b611-1f129ccc4f69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1301e352-zpl5b" Oct 30 20:39:24.636229 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:24.636200 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ac38922-9d8d-4005-b611-1f129ccc4f69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1301e352-zpl5b\" (UID: \"9ac38922-9d8d-4005-b611-1f129ccc4f69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1301e352-zpl5b" Oct 30 20:39:24.636905 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:24.636883 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ac38922-9d8d-4005-b611-1f129ccc4f69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1301e352-zpl5b\" (UID: \"9ac38922-9d8d-4005-b611-1f129ccc4f69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1301e352-zpl5b" Oct 30 20:39:24.638413 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:24.638396 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ac38922-9d8d-4005-b611-1f129ccc4f69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1301e352-zpl5b\" (UID: \"9ac38922-9d8d-4005-b611-1f129ccc4f69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1301e352-zpl5b" Oct 30 20:39:24.656720 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:24.656696 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-997c20ab-h8thk" event={"ID":"160c9418-9850-4c39-8691-55202031b0a4","Type":"ContainerStarted","Data":"d964b4e29f1b38b257693ce7d7c85f7e549584b99d95c19686eaaaecb6bce758"} Oct 30 20:39:24.698939 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:24.698916 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54spr\" (UniqueName: \"kubernetes.io/projected/9ac38922-9d8d-4005-b611-1f129ccc4f69-kube-api-access-54spr\") pod \"maas-api-revoke-sub-1301e352-zpl5b\" (UID: \"9ac38922-9d8d-4005-b611-1f129ccc4f69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1301e352-zpl5b" Oct 30 20:39:24.767137 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:24.767113 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1301e352-zpl5b" Oct 30 20:39:24.965478 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:24.965456 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b75042a-jbssd"] Oct 30 20:39:24.967337 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:39:24.967316 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30ae8747_6c31_4160_8548_1a80e5305294.slice/crio-434738c5117503bb1ba27f2b799143ccd0db211188b51d27ff1700898ae475a8 WatchSource:0}: Error finding container 434738c5117503bb1ba27f2b799143ccd0db211188b51d27ff1700898ae475a8: Status 404 returned error can't find the container with id 434738c5117503bb1ba27f2b799143ccd0db211188b51d27ff1700898ae475a8 Oct 30 20:39:25.152700 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:25.152677 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1301e352-zpl5b"] Oct 30 20:39:25.155173 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:39:25.155146 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ac38922_9d8d_4005_b611_1f129ccc4f69.slice/crio-1ec531109af4ab5e3607b6054073c7a2e98e5b2edbadb92bb0ab9f530bcfb144 WatchSource:0}: Error finding container 1ec531109af4ab5e3607b6054073c7a2e98e5b2edbadb92bb0ab9f530bcfb144: Status 404 returned error can't find the container with id 1ec531109af4ab5e3607b6054073c7a2e98e5b2edbadb92bb0ab9f530bcfb144 Oct 30 20:39:25.666784 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:25.666697 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1301e352-zpl5b" event={"ID":"9ac38922-9d8d-4005-b611-1f129ccc4f69","Type":"ContainerStarted","Data":"e8d42dda38cba8f661981fe0ed3639f1fd9b3a90801cbfe67125d1721df061a7"} Oct 30 20:39:25.666784 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:25.666756 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1301e352-zpl5b" event={"ID":"9ac38922-9d8d-4005-b611-1f129ccc4f69","Type":"ContainerStarted","Data":"1ec531109af4ab5e3607b6054073c7a2e98e5b2edbadb92bb0ab9f530bcfb144"} Oct 30 20:39:25.668588 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:25.668562 4292 generic.go:356] "Generic (PLEG): container finished" podID="160c9418-9850-4c39-8691-55202031b0a4" containerID="82306d7f3221f05b027c601e16a6f300999c96706388e00b44628464a0194269" exitCode=0 Oct 30 20:39:25.668748 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:25.668629 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-997c20ab-h8thk" event={"ID":"160c9418-9850-4c39-8691-55202031b0a4","Type":"ContainerDied","Data":"82306d7f3221f05b027c601e16a6f300999c96706388e00b44628464a0194269"} Oct 30 20:39:25.670035 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:25.670014 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b75042a-jbssd" event={"ID":"30ae8747-6c31-4160-8548-1a80e5305294","Type":"ContainerStarted","Data":"4fc70b62e530cc09df4926fcd2b7daf997cda443b297b117b8393e2cf9b40984"} Oct 30 20:39:25.670105 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:25.670051 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b75042a-jbssd" event={"ID":"30ae8747-6c31-4160-8548-1a80e5305294","Type":"ContainerStarted","Data":"434738c5117503bb1ba27f2b799143ccd0db211188b51d27ff1700898ae475a8"} Oct 30 20:39:25.787227 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:25.787174 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1301e352-zpl5b" podStartSLOduration=1.787161732 podStartE2EDuration="1.787161732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:39:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:39:25.785381388 +0000 UTC m=+4469.303696867" watchObservedRunningTime="2026-10-30 20:39:25.787161732 +0000 UTC m=+4469.305477258" Oct 30 20:39:26.025267 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:26.025215 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b75042a-jbssd" podStartSLOduration=3.025202463 podStartE2EDuration="3.025202463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:39:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:39:26.024549647 +0000 UTC m=+4469.542865094" watchObservedRunningTime="2026-10-30 20:39:26.025202463 +0000 UTC m=+4469.543517917" Oct 30 20:39:26.676590 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:26.676557 4292 generic.go:356] "Generic (PLEG): container finished" podID="30ae8747-6c31-4160-8548-1a80e5305294" containerID="4fc70b62e530cc09df4926fcd2b7daf997cda443b297b117b8393e2cf9b40984" exitCode=0 Oct 30 20:39:26.676774 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:26.676642 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b75042a-jbssd" event={"ID":"30ae8747-6c31-4160-8548-1a80e5305294","Type":"ContainerDied","Data":"4fc70b62e530cc09df4926fcd2b7daf997cda443b297b117b8393e2cf9b40984"} Oct 30 20:39:26.678187 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:26.678160 4292 generic.go:356] "Generic (PLEG): container finished" podID="9ac38922-9d8d-4005-b611-1f129ccc4f69" containerID="e8d42dda38cba8f661981fe0ed3639f1fd9b3a90801cbfe67125d1721df061a7" exitCode=0 Oct 30 20:39:26.678375 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:26.678223 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1301e352-zpl5b" event={"ID":"9ac38922-9d8d-4005-b611-1f129ccc4f69","Type":"ContainerDied","Data":"e8d42dda38cba8f661981fe0ed3639f1fd9b3a90801cbfe67125d1721df061a7"} Oct 30 20:39:26.804255 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:26.804230 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-997c20ab-h8thk" Oct 30 20:39:26.958810 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:26.958736 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/160c9418-9850-4c39-8691-55202031b0a4-kube-api-access-tdqql\" (UniqueName: \"kubernetes.io/projected/160c9418-9850-4c39-8691-55202031b0a4-kube-api-access-tdqql\") pod \"160c9418-9850-4c39-8691-55202031b0a4\" (UID: \"160c9418-9850-4c39-8691-55202031b0a4\") " Oct 30 20:39:26.958810 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:26.958781 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/160c9418-9850-4c39-8691-55202031b0a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/160c9418-9850-4c39-8691-55202031b0a4-maas-api-xks-ca\") pod \"160c9418-9850-4c39-8691-55202031b0a4\" (UID: \"160c9418-9850-4c39-8691-55202031b0a4\") " Oct 30 20:39:26.958993 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:26.958818 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/160c9418-9850-4c39-8691-55202031b0a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/160c9418-9850-4c39-8691-55202031b0a4-maas-api-openshift-ca\") pod \"160c9418-9850-4c39-8691-55202031b0a4\" (UID: \"160c9418-9850-4c39-8691-55202031b0a4\") " Oct 30 20:39:26.959260 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:26.959235 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/160c9418-9850-4c39-8691-55202031b0a4-maas-api-openshift-ca" pod "160c9418-9850-4c39-8691-55202031b0a4" (UID: "160c9418-9850-4c39-8691-55202031b0a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:39:26.960759 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:26.960739 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/160c9418-9850-4c39-8691-55202031b0a4-maas-api-xks-ca" pod "160c9418-9850-4c39-8691-55202031b0a4" (UID: "160c9418-9850-4c39-8691-55202031b0a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:39:26.960869 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:26.960759 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/160c9418-9850-4c39-8691-55202031b0a4-kube-api-access-tdqql" pod "160c9418-9850-4c39-8691-55202031b0a4" (UID: "160c9418-9850-4c39-8691-55202031b0a4"). InnerVolumeSpecName "kube-api-access-tdqql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:39:27.059758 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:27.059727 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/160c9418-9850-4c39-8691-55202031b0a4-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:39:27.059758 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:27.059755 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/160c9418-9850-4c39-8691-55202031b0a4-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:39:27.060142 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:27.059766 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdqql\" (UniqueName: \"kubernetes.io/projected/160c9418-9850-4c39-8691-55202031b0a4-kube-api-access-tdqql\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:39:27.685473 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:27.685434 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-997c20ab-h8thk" event={"ID":"160c9418-9850-4c39-8691-55202031b0a4","Type":"ContainerDied","Data":"d964b4e29f1b38b257693ce7d7c85f7e549584b99d95c19686eaaaecb6bce758"} Oct 30 20:39:27.685671 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:27.685484 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d964b4e29f1b38b257693ce7d7c85f7e549584b99d95c19686eaaaecb6bce758" Oct 30 20:39:27.685671 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:27.685564 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-997c20ab-h8thk" Oct 30 20:39:27.820007 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:27.819983 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1301e352-zpl5b" Oct 30 20:39:27.845899 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:27.845880 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b75042a-jbssd" Oct 30 20:39:27.968237 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:27.968164 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30ae8747-6c31-4160-8548-1a80e5305294-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30ae8747-6c31-4160-8548-1a80e5305294-maas-api-openshift-ca\") pod \"30ae8747-6c31-4160-8548-1a80e5305294\" (UID: \"30ae8747-6c31-4160-8548-1a80e5305294\") " Oct 30 20:39:27.968415 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:27.968249 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30ae8747-6c31-4160-8548-1a80e5305294-kube-api-access-4nslq\" (UniqueName: \"kubernetes.io/projected/30ae8747-6c31-4160-8548-1a80e5305294-kube-api-access-4nslq\") pod \"30ae8747-6c31-4160-8548-1a80e5305294\" (UID: \"30ae8747-6c31-4160-8548-1a80e5305294\") " Oct 30 20:39:27.968415 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:27.968304 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30ae8747-6c31-4160-8548-1a80e5305294-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30ae8747-6c31-4160-8548-1a80e5305294-maas-api-xks-ca\") pod \"30ae8747-6c31-4160-8548-1a80e5305294\" (UID: \"30ae8747-6c31-4160-8548-1a80e5305294\") " Oct 30 20:39:27.968415 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:27.968341 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ac38922-9d8d-4005-b611-1f129ccc4f69-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ac38922-9d8d-4005-b611-1f129ccc4f69-maas-api-xks-ca\") pod \"9ac38922-9d8d-4005-b611-1f129ccc4f69\" (UID: \"9ac38922-9d8d-4005-b611-1f129ccc4f69\") " Oct 30 20:39:27.968415 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:27.968390 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ac38922-9d8d-4005-b611-1f129ccc4f69-kube-api-access-54spr\" (UniqueName: \"kubernetes.io/projected/9ac38922-9d8d-4005-b611-1f129ccc4f69-kube-api-access-54spr\") pod \"9ac38922-9d8d-4005-b611-1f129ccc4f69\" (UID: \"9ac38922-9d8d-4005-b611-1f129ccc4f69\") " Oct 30 20:39:27.968646 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:27.968469 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ac38922-9d8d-4005-b611-1f129ccc4f69-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ac38922-9d8d-4005-b611-1f129ccc4f69-maas-api-openshift-ca\") pod \"9ac38922-9d8d-4005-b611-1f129ccc4f69\" (UID: \"9ac38922-9d8d-4005-b611-1f129ccc4f69\") " Oct 30 20:39:27.968646 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:27.968553 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30ae8747-6c31-4160-8548-1a80e5305294-maas-api-openshift-ca" pod "30ae8747-6c31-4160-8548-1a80e5305294" (UID: "30ae8747-6c31-4160-8548-1a80e5305294"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:39:27.968749 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:27.968738 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30ae8747-6c31-4160-8548-1a80e5305294-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:39:27.968945 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:27.968918 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ac38922-9d8d-4005-b611-1f129ccc4f69-maas-api-openshift-ca" pod "9ac38922-9d8d-4005-b611-1f129ccc4f69" (UID: "9ac38922-9d8d-4005-b611-1f129ccc4f69"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:39:27.970647 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:27.970561 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30ae8747-6c31-4160-8548-1a80e5305294-kube-api-access-4nslq" pod "30ae8747-6c31-4160-8548-1a80e5305294" (UID: "30ae8747-6c31-4160-8548-1a80e5305294"). InnerVolumeSpecName "kube-api-access-4nslq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:39:27.970735 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:27.970673 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ac38922-9d8d-4005-b611-1f129ccc4f69-maas-api-xks-ca" pod "9ac38922-9d8d-4005-b611-1f129ccc4f69" (UID: "9ac38922-9d8d-4005-b611-1f129ccc4f69"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:39:27.970970 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:27.970947 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30ae8747-6c31-4160-8548-1a80e5305294-maas-api-xks-ca" pod "30ae8747-6c31-4160-8548-1a80e5305294" (UID: "30ae8747-6c31-4160-8548-1a80e5305294"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:39:27.971068 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:27.971007 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ac38922-9d8d-4005-b611-1f129ccc4f69-kube-api-access-54spr" pod "9ac38922-9d8d-4005-b611-1f129ccc4f69" (UID: "9ac38922-9d8d-4005-b611-1f129ccc4f69"). InnerVolumeSpecName "kube-api-access-54spr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:39:28.069744 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:28.069712 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nslq\" (UniqueName: \"kubernetes.io/projected/30ae8747-6c31-4160-8548-1a80e5305294-kube-api-access-4nslq\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:39:28.069744 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:28.069741 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30ae8747-6c31-4160-8548-1a80e5305294-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:39:28.070130 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:28.069751 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ac38922-9d8d-4005-b611-1f129ccc4f69-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:39:28.070130 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:28.069761 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54spr\" (UniqueName: \"kubernetes.io/projected/9ac38922-9d8d-4005-b611-1f129ccc4f69-kube-api-access-54spr\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:39:28.070130 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:28.069770 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ac38922-9d8d-4005-b611-1f129ccc4f69-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:39:28.693236 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:28.693205 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b75042a-jbssd" Oct 30 20:39:28.693236 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:28.693223 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b75042a-jbssd" event={"ID":"30ae8747-6c31-4160-8548-1a80e5305294","Type":"ContainerDied","Data":"434738c5117503bb1ba27f2b799143ccd0db211188b51d27ff1700898ae475a8"} Oct 30 20:39:28.693497 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:28.693274 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="434738c5117503bb1ba27f2b799143ccd0db211188b51d27ff1700898ae475a8" Oct 30 20:39:28.694590 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:28.694554 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1301e352-zpl5b" Oct 30 20:39:28.694738 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:28.694599 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1301e352-zpl5b" event={"ID":"9ac38922-9d8d-4005-b611-1f129ccc4f69","Type":"ContainerDied","Data":"1ec531109af4ab5e3607b6054073c7a2e98e5b2edbadb92bb0ab9f530bcfb144"} Oct 30 20:39:28.694738 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:28.694635 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ec531109af4ab5e3607b6054073c7a2e98e5b2edbadb92bb0ab9f530bcfb144" Oct 30 20:39:36.859112 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:36.859073 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-273c1b/payload-processing-external-model-e2e-extmod-b-273c1b-5666q2xkc"] Oct 30 20:39:41.860471 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:41.860377 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-273c1b/payload-processing-external-model-e2e-extmod-b-273c1b-5666q2xkc" podUID="94ce740c-20b4-4cd5-b060-898e41e38f62" containerName="payload-processing" containerID="cri-o://db00c143e2317049cc1732e5a718887cdfc7c1a7ec0e34fa3d4811cf5dcecfd6" gracePeriod=25 Oct 30 20:39:42.098369 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:42.098327 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-273c1b/payload-processing-external-model-e2e-extmod-b-273c1b-5666q2xkc" Oct 30 20:39:42.184889 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:42.184820 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94ce740c-20b4-4cd5-b060-898e41e38f62-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94ce740c-20b4-4cd5-b060-898e41e38f62-plugins-config-volume\") pod \"94ce740c-20b4-4cd5-b060-898e41e38f62\" (UID: \"94ce740c-20b4-4cd5-b060-898e41e38f62\") " Oct 30 20:39:42.184889 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:42.184854 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94ce740c-20b4-4cd5-b060-898e41e38f62-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/94ce740c-20b4-4cd5-b060-898e41e38f62-routing-overlay-volume\") pod \"94ce740c-20b4-4cd5-b060-898e41e38f62\" (UID: \"94ce740c-20b4-4cd5-b060-898e41e38f62\") " Oct 30 20:39:42.184889 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:42.184888 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94ce740c-20b4-4cd5-b060-898e41e38f62-provider-credentials\" (UniqueName: \"kubernetes.io/projected/94ce740c-20b4-4cd5-b060-898e41e38f62-provider-credentials\") pod \"94ce740c-20b4-4cd5-b060-898e41e38f62\" (UID: \"94ce740c-20b4-4cd5-b060-898e41e38f62\") " Oct 30 20:39:42.185106 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:42.184926 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94ce740c-20b4-4cd5-b060-898e41e38f62-kube-api-access-9zss5\" (UniqueName: \"kubernetes.io/projected/94ce740c-20b4-4cd5-b060-898e41e38f62-kube-api-access-9zss5\") pod \"94ce740c-20b4-4cd5-b060-898e41e38f62\" (UID: \"94ce740c-20b4-4cd5-b060-898e41e38f62\") " Oct 30 20:39:42.185255 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:42.185232 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94ce740c-20b4-4cd5-b060-898e41e38f62-routing-overlay-volume" pod "94ce740c-20b4-4cd5-b060-898e41e38f62" (UID: "94ce740c-20b4-4cd5-b060-898e41e38f62"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:39:42.185308 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:42.185268 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94ce740c-20b4-4cd5-b060-898e41e38f62-plugins-config-volume" pod "94ce740c-20b4-4cd5-b060-898e41e38f62" (UID: "94ce740c-20b4-4cd5-b060-898e41e38f62"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:39:42.187052 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:42.187004 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94ce740c-20b4-4cd5-b060-898e41e38f62-provider-credentials" pod "94ce740c-20b4-4cd5-b060-898e41e38f62" (UID: "94ce740c-20b4-4cd5-b060-898e41e38f62"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:39:42.187052 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:42.187032 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94ce740c-20b4-4cd5-b060-898e41e38f62-kube-api-access-9zss5" pod "94ce740c-20b4-4cd5-b060-898e41e38f62" (UID: "94ce740c-20b4-4cd5-b060-898e41e38f62"). InnerVolumeSpecName "kube-api-access-9zss5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:39:42.286653 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:42.286622 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94ce740c-20b4-4cd5-b060-898e41e38f62-plugins-config-volume\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:39:42.286653 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:42.286650 4292 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/94ce740c-20b4-4cd5-b060-898e41e38f62-routing-overlay-volume\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:39:42.286653 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:42.286661 4292 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/94ce740c-20b4-4cd5-b060-898e41e38f62-provider-credentials\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:39:42.286874 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:42.286671 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zss5\" (UniqueName: \"kubernetes.io/projected/94ce740c-20b4-4cd5-b060-898e41e38f62-kube-api-access-9zss5\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:39:42.764390 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:42.764346 4292 generic.go:356] "Generic (PLEG): container finished" podID="94ce740c-20b4-4cd5-b060-898e41e38f62" containerID="db00c143e2317049cc1732e5a718887cdfc7c1a7ec0e34fa3d4811cf5dcecfd6" exitCode=0 Oct 30 20:39:42.764637 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:42.764415 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-273c1b/payload-processing-external-model-e2e-extmod-b-273c1b-5666q2xkc" Oct 30 20:39:42.764637 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:42.764418 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-273c1b/payload-processing-external-model-e2e-extmod-b-273c1b-5666q2xkc" event={"ID":"94ce740c-20b4-4cd5-b060-898e41e38f62","Type":"ContainerDied","Data":"db00c143e2317049cc1732e5a718887cdfc7c1a7ec0e34fa3d4811cf5dcecfd6"} Oct 30 20:39:42.764637 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:42.764532 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-273c1b/payload-processing-external-model-e2e-extmod-b-273c1b-5666q2xkc" event={"ID":"94ce740c-20b4-4cd5-b060-898e41e38f62","Type":"ContainerDied","Data":"de6decd8ec7e56dc109bac8e8a9226e33ab5731d5f28d7054963f4012dd8df46"} Oct 30 20:39:42.764637 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:42.764555 4292 scope.go:122] "RemoveContainer" containerID="db00c143e2317049cc1732e5a718887cdfc7c1a7ec0e34fa3d4811cf5dcecfd6" Oct 30 20:39:42.772927 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:42.772911 4292 scope.go:122] "RemoveContainer" containerID="db00c143e2317049cc1732e5a718887cdfc7c1a7ec0e34fa3d4811cf5dcecfd6" Oct 30 20:39:42.773212 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:39:42.773195 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db00c143e2317049cc1732e5a718887cdfc7c1a7ec0e34fa3d4811cf5dcecfd6\": container with ID starting with db00c143e2317049cc1732e5a718887cdfc7c1a7ec0e34fa3d4811cf5dcecfd6 not found: ID does not exist" containerID="db00c143e2317049cc1732e5a718887cdfc7c1a7ec0e34fa3d4811cf5dcecfd6" Oct 30 20:39:42.773294 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:42.773220 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db00c143e2317049cc1732e5a718887cdfc7c1a7ec0e34fa3d4811cf5dcecfd6"} err="failed to get container status \"db00c143e2317049cc1732e5a718887cdfc7c1a7ec0e34fa3d4811cf5dcecfd6\": rpc error: code = NotFound desc = could not find container \"db00c143e2317049cc1732e5a718887cdfc7c1a7ec0e34fa3d4811cf5dcecfd6\": container with ID starting with db00c143e2317049cc1732e5a718887cdfc7c1a7ec0e34fa3d4811cf5dcecfd6 not found: ID does not exist" Oct 30 20:39:42.934748 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:42.934671 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-273c1b/payload-processing-external-model-e2e-extmod-b-273c1b-5666q2xkc"] Oct 30 20:39:42.985645 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:42.985618 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-273c1b/payload-processing-external-model-e2e-extmod-b-273c1b-5666q2xkc"] Oct 30 20:39:43.219113 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:39:43.219027 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:39:44.581227 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:44.581190 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94ce740c-20b4-4cd5-b060-898e41e38f62" path="/var/lib/kubelet/pods/94ce740c-20b4-4cd5-b060-898e41e38f62/volumes" Oct 30 20:39:53.433271 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:53.433235 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-46dadd-cm5bw"] Oct 30 20:39:53.433713 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:53.433598 4292 cpu_manager.go:414] "containerMap: removing container" podUID="30ae8747-6c31-4160-8548-1a80e5305294" containerName="revoke-keys" Oct 30 20:39:53.433713 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:53.433610 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30ae8747-6c31-4160-8548-1a80e5305294" containerName="revoke-keys" Oct 30 20:39:53.433713 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:53.433623 4292 cpu_manager.go:414] "containerMap: removing container" podUID="94ce740c-20b4-4cd5-b060-898e41e38f62" containerName="payload-processing" Oct 30 20:39:53.433713 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:53.433628 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94ce740c-20b4-4cd5-b060-898e41e38f62" containerName="payload-processing" Oct 30 20:39:53.433713 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:53.433640 4292 cpu_manager.go:414] "containerMap: removing container" podUID="9ac38922-9d8d-4005-b611-1f129ccc4f69" containerName="revoke-keys" Oct 30 20:39:53.433713 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:53.433645 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ac38922-9d8d-4005-b611-1f129ccc4f69" containerName="revoke-keys" Oct 30 20:39:53.433713 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:53.433665 4292 cpu_manager.go:414] "containerMap: removing container" podUID="160c9418-9850-4c39-8691-55202031b0a4" containerName="revoke-keys" Oct 30 20:39:53.433713 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:53.433669 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="160c9418-9850-4c39-8691-55202031b0a4" containerName="revoke-keys" Oct 30 20:39:53.433713 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:53.433716 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="160c9418-9850-4c39-8691-55202031b0a4" containerName="revoke-keys" Oct 30 20:39:53.434010 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:53.433725 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="30ae8747-6c31-4160-8548-1a80e5305294" containerName="revoke-keys" Oct 30 20:39:53.434010 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:53.433731 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="94ce740c-20b4-4cd5-b060-898e41e38f62" containerName="payload-processing" Oct 30 20:39:53.434010 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:53.433737 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ac38922-9d8d-4005-b611-1f129ccc4f69" containerName="revoke-keys" Oct 30 20:39:53.437939 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:53.437922 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-46dadd-cm5bw" Oct 30 20:39:53.488626 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:53.488596 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvn7w\" (UniqueName: \"kubernetes.io/projected/3839d17e-da15-442e-a305-54e33afa5bad-kube-api-access-xvn7w\") pod \"maas-api-revoke-keys-e2e-extmod-a-46dadd-cm5bw\" (UID: \"3839d17e-da15-442e-a305-54e33afa5bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-46dadd-cm5bw" Oct 30 20:39:53.488753 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:53.488629 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3839d17e-da15-442e-a305-54e33afa5bad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-46dadd-cm5bw\" (UID: \"3839d17e-da15-442e-a305-54e33afa5bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-46dadd-cm5bw" Oct 30 20:39:53.488753 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:53.488658 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3839d17e-da15-442e-a305-54e33afa5bad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-46dadd-cm5bw\" (UID: \"3839d17e-da15-442e-a305-54e33afa5bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-46dadd-cm5bw" Oct 30 20:39:53.498477 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:53.498449 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-46dadd-cm5bw"] Oct 30 20:39:53.546487 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:53.546437 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:39:53.546649 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:53.546511 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\"" Oct 30 20:39:53.589810 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:53.589780 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3839d17e-da15-442e-a305-54e33afa5bad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-46dadd-cm5bw\" (UID: \"3839d17e-da15-442e-a305-54e33afa5bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-46dadd-cm5bw" Oct 30 20:39:53.589952 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:53.589872 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvn7w\" (UniqueName: \"kubernetes.io/projected/3839d17e-da15-442e-a305-54e33afa5bad-kube-api-access-xvn7w\") pod \"maas-api-revoke-keys-e2e-extmod-a-46dadd-cm5bw\" (UID: \"3839d17e-da15-442e-a305-54e33afa5bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-46dadd-cm5bw" Oct 30 20:39:53.589952 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:53.589897 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3839d17e-da15-442e-a305-54e33afa5bad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-46dadd-cm5bw\" (UID: \"3839d17e-da15-442e-a305-54e33afa5bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-46dadd-cm5bw" Oct 30 20:39:53.590444 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:53.590426 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3839d17e-da15-442e-a305-54e33afa5bad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-46dadd-cm5bw\" (UID: \"3839d17e-da15-442e-a305-54e33afa5bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-46dadd-cm5bw" Oct 30 20:39:53.592127 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:53.592111 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3839d17e-da15-442e-a305-54e33afa5bad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-46dadd-cm5bw\" (UID: \"3839d17e-da15-442e-a305-54e33afa5bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-46dadd-cm5bw" Oct 30 20:39:53.659852 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:53.659825 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvn7w\" (UniqueName: \"kubernetes.io/projected/3839d17e-da15-442e-a305-54e33afa5bad-kube-api-access-xvn7w\") pod \"maas-api-revoke-keys-e2e-extmod-a-46dadd-cm5bw\" (UID: \"3839d17e-da15-442e-a305-54e33afa5bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-46dadd-cm5bw" Oct 30 20:39:53.749953 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:53.749880 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-46dadd-cm5bw" Oct 30 20:39:54.142500 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:54.140715 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-46dadd-cm5bw"] Oct 30 20:39:54.143143 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:39:54.143111 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3839d17e_da15_442e_a305_54e33afa5bad.slice/crio-5fd2810410b958727629693280d239c45b4143046365e62782c3b757ce530130 WatchSource:0}: Error finding container 5fd2810410b958727629693280d239c45b4143046365e62782c3b757ce530130: Status 404 returned error can't find the container with id 5fd2810410b958727629693280d239c45b4143046365e62782c3b757ce530130 Oct 30 20:39:54.832847 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:54.832811 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-46dadd-cm5bw" event={"ID":"3839d17e-da15-442e-a305-54e33afa5bad","Type":"ContainerStarted","Data":"c95975209fa982b0f807297500a4fff0b597b640b1f420bece55d65925c3e0ae"} Oct 30 20:39:54.833248 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:54.832872 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-46dadd-cm5bw" event={"ID":"3839d17e-da15-442e-a305-54e33afa5bad","Type":"ContainerStarted","Data":"5fd2810410b958727629693280d239c45b4143046365e62782c3b757ce530130"} Oct 30 20:39:54.952943 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:54.952891 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-46dadd-cm5bw" podStartSLOduration=1.952875589 podStartE2EDuration="1.952875589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:39:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:39:54.951839044 +0000 UTC m=+4498.470154500" watchObservedRunningTime="2026-10-30 20:39:54.952875589 +0000 UTC m=+4498.471191045" Oct 30 20:39:55.839590 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:55.839558 4292 generic.go:356] "Generic (PLEG): container finished" podID="3839d17e-da15-442e-a305-54e33afa5bad" containerID="c95975209fa982b0f807297500a4fff0b597b640b1f420bece55d65925c3e0ae" exitCode=0 Oct 30 20:39:55.839998 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:55.839646 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-46dadd-cm5bw" event={"ID":"3839d17e-da15-442e-a305-54e33afa5bad","Type":"ContainerDied","Data":"c95975209fa982b0f807297500a4fff0b597b640b1f420bece55d65925c3e0ae"} Oct 30 20:39:56.974072 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:56.974047 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-46dadd-cm5bw" Oct 30 20:39:57.019180 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:57.019150 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3839d17e-da15-442e-a305-54e33afa5bad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3839d17e-da15-442e-a305-54e33afa5bad-maas-api-xks-ca\") pod \"3839d17e-da15-442e-a305-54e33afa5bad\" (UID: \"3839d17e-da15-442e-a305-54e33afa5bad\") " Oct 30 20:39:57.019316 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:57.019208 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3839d17e-da15-442e-a305-54e33afa5bad-kube-api-access-xvn7w\" (UniqueName: \"kubernetes.io/projected/3839d17e-da15-442e-a305-54e33afa5bad-kube-api-access-xvn7w\") pod \"3839d17e-da15-442e-a305-54e33afa5bad\" (UID: \"3839d17e-da15-442e-a305-54e33afa5bad\") " Oct 30 20:39:57.019316 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:57.019252 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3839d17e-da15-442e-a305-54e33afa5bad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3839d17e-da15-442e-a305-54e33afa5bad-maas-api-openshift-ca\") pod \"3839d17e-da15-442e-a305-54e33afa5bad\" (UID: \"3839d17e-da15-442e-a305-54e33afa5bad\") " Oct 30 20:39:57.019677 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:57.019651 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3839d17e-da15-442e-a305-54e33afa5bad-maas-api-openshift-ca" pod "3839d17e-da15-442e-a305-54e33afa5bad" (UID: "3839d17e-da15-442e-a305-54e33afa5bad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:39:57.021189 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:57.021172 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3839d17e-da15-442e-a305-54e33afa5bad-kube-api-access-xvn7w" pod "3839d17e-da15-442e-a305-54e33afa5bad" (UID: "3839d17e-da15-442e-a305-54e33afa5bad"). InnerVolumeSpecName "kube-api-access-xvn7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:39:57.021286 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:57.021257 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3839d17e-da15-442e-a305-54e33afa5bad-maas-api-xks-ca" pod "3839d17e-da15-442e-a305-54e33afa5bad" (UID: "3839d17e-da15-442e-a305-54e33afa5bad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:39:57.120993 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:57.120907 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3839d17e-da15-442e-a305-54e33afa5bad-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:39:57.120993 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:57.120938 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvn7w\" (UniqueName: \"kubernetes.io/projected/3839d17e-da15-442e-a305-54e33afa5bad-kube-api-access-xvn7w\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:39:57.120993 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:57.120955 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3839d17e-da15-442e-a305-54e33afa5bad-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:39:57.856411 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:57.856375 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-46dadd-cm5bw" Oct 30 20:39:57.856604 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:57.856378 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-46dadd-cm5bw" event={"ID":"3839d17e-da15-442e-a305-54e33afa5bad","Type":"ContainerDied","Data":"5fd2810410b958727629693280d239c45b4143046365e62782c3b757ce530130"} Oct 30 20:39:57.856604 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:39:57.856534 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fd2810410b958727629693280d239c45b4143046365e62782c3b757ce530130" Oct 30 20:40:02.851687 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:02.851653 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a74d15-vl7fc"] Oct 30 20:40:02.902905 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:02.902878 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a74d15-vl7fc"] Oct 30 20:40:04.580591 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:04.580561 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fe8fba7-ab2b-4b29-8c7c-9999ca750a97" path="/var/lib/kubelet/pods/4fe8fba7-ab2b-4b29-8c7c-9999ca750a97/volumes" Oct 30 20:40:13.219262 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:40:13.219227 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:40:24.452609 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:24.452568 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-46dadd-cm5bw"] Oct 30 20:40:24.503079 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:24.503047 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-46dadd-cm5bw"] Oct 30 20:40:24.582076 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:24.582045 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3839d17e-da15-442e-a305-54e33afa5bad" path="/var/lib/kubelet/pods/3839d17e-da15-442e-a305-54e33afa5bad/volumes" Oct 30 20:40:32.432053 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:32.432021 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-273c1b-hwpg5"] Oct 30 20:40:32.432580 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:32.432555 4292 cpu_manager.go:414] "containerMap: removing container" podUID="3839d17e-da15-442e-a305-54e33afa5bad" containerName="revoke-keys" Oct 30 20:40:32.432580 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:32.432575 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3839d17e-da15-442e-a305-54e33afa5bad" containerName="revoke-keys" Oct 30 20:40:32.432702 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:32.432671 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="3839d17e-da15-442e-a305-54e33afa5bad" containerName="revoke-keys" Oct 30 20:40:32.436008 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:32.435986 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-273c1b-hwpg5" Oct 30 20:40:32.489710 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:32.489678 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-273c1b-hwpg5"] Oct 30 20:40:32.538621 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:32.538592 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:40:32.538791 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:32.538654 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\"" Oct 30 20:40:32.541015 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:32.540997 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a3b6aaf-4347-4b64-8681-e1218d8f5422-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-273c1b-hwpg5\" (UID: \"1a3b6aaf-4347-4b64-8681-e1218d8f5422\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-273c1b-hwpg5" Oct 30 20:40:32.541112 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:32.541035 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtvv6\" (UniqueName: \"kubernetes.io/projected/1a3b6aaf-4347-4b64-8681-e1218d8f5422-kube-api-access-xtvv6\") pod \"maas-api-revoke-keys-e2e-extmod-b-273c1b-hwpg5\" (UID: \"1a3b6aaf-4347-4b64-8681-e1218d8f5422\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-273c1b-hwpg5" Oct 30 20:40:32.541159 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:32.541146 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a3b6aaf-4347-4b64-8681-e1218d8f5422-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-273c1b-hwpg5\" (UID: \"1a3b6aaf-4347-4b64-8681-e1218d8f5422\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-273c1b-hwpg5" Oct 30 20:40:32.642137 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:32.642099 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a3b6aaf-4347-4b64-8681-e1218d8f5422-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-273c1b-hwpg5\" (UID: \"1a3b6aaf-4347-4b64-8681-e1218d8f5422\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-273c1b-hwpg5" Oct 30 20:40:32.642422 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:32.642153 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtvv6\" (UniqueName: \"kubernetes.io/projected/1a3b6aaf-4347-4b64-8681-e1218d8f5422-kube-api-access-xtvv6\") pod \"maas-api-revoke-keys-e2e-extmod-b-273c1b-hwpg5\" (UID: \"1a3b6aaf-4347-4b64-8681-e1218d8f5422\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-273c1b-hwpg5" Oct 30 20:40:32.642422 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:32.642279 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a3b6aaf-4347-4b64-8681-e1218d8f5422-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-273c1b-hwpg5\" (UID: \"1a3b6aaf-4347-4b64-8681-e1218d8f5422\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-273c1b-hwpg5" Oct 30 20:40:32.642952 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:32.642928 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a3b6aaf-4347-4b64-8681-e1218d8f5422-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-273c1b-hwpg5\" (UID: \"1a3b6aaf-4347-4b64-8681-e1218d8f5422\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-273c1b-hwpg5" Oct 30 20:40:32.644586 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:32.644566 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a3b6aaf-4347-4b64-8681-e1218d8f5422-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-273c1b-hwpg5\" (UID: \"1a3b6aaf-4347-4b64-8681-e1218d8f5422\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-273c1b-hwpg5" Oct 30 20:40:32.705532 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:32.705472 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtvv6\" (UniqueName: \"kubernetes.io/projected/1a3b6aaf-4347-4b64-8681-e1218d8f5422-kube-api-access-xtvv6\") pod \"maas-api-revoke-keys-e2e-extmod-b-273c1b-hwpg5\" (UID: \"1a3b6aaf-4347-4b64-8681-e1218d8f5422\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-273c1b-hwpg5" Oct 30 20:40:32.748152 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:32.748127 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-273c1b-hwpg5" Oct 30 20:40:33.122381 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:33.122339 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-273c1b-hwpg5"] Oct 30 20:40:33.124017 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:40:33.123988 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a3b6aaf_4347_4b64_8681_e1218d8f5422.slice/crio-7efb8723522a22761882ac83e73dbbcde108752c9f66196b3fa7cb280d49d73d WatchSource:0}: Error finding container 7efb8723522a22761882ac83e73dbbcde108752c9f66196b3fa7cb280d49d73d: Status 404 returned error can't find the container with id 7efb8723522a22761882ac83e73dbbcde108752c9f66196b3fa7cb280d49d73d Oct 30 20:40:34.037832 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:34.037799 4292 generic.go:356] "Generic (PLEG): container finished" podID="1a3b6aaf-4347-4b64-8681-e1218d8f5422" containerID="b1c04edf7bf460d05d7407fd7b7e65533286955aa3d25bdd913b129b4e61ea31" exitCode=0 Oct 30 20:40:34.038158 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:34.037880 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-273c1b-hwpg5" event={"ID":"1a3b6aaf-4347-4b64-8681-e1218d8f5422","Type":"ContainerDied","Data":"b1c04edf7bf460d05d7407fd7b7e65533286955aa3d25bdd913b129b4e61ea31"} Oct 30 20:40:34.038158 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:34.037934 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-273c1b-hwpg5" event={"ID":"1a3b6aaf-4347-4b64-8681-e1218d8f5422","Type":"ContainerStarted","Data":"7efb8723522a22761882ac83e73dbbcde108752c9f66196b3fa7cb280d49d73d"} Oct 30 20:40:35.179387 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:35.179359 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-273c1b-hwpg5" Oct 30 20:40:35.264385 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:35.264332 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a3b6aaf-4347-4b64-8681-e1218d8f5422-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a3b6aaf-4347-4b64-8681-e1218d8f5422-maas-api-xks-ca\") pod \"1a3b6aaf-4347-4b64-8681-e1218d8f5422\" (UID: \"1a3b6aaf-4347-4b64-8681-e1218d8f5422\") " Oct 30 20:40:35.264632 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:35.264399 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a3b6aaf-4347-4b64-8681-e1218d8f5422-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a3b6aaf-4347-4b64-8681-e1218d8f5422-maas-api-openshift-ca\") pod \"1a3b6aaf-4347-4b64-8681-e1218d8f5422\" (UID: \"1a3b6aaf-4347-4b64-8681-e1218d8f5422\") " Oct 30 20:40:35.264632 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:35.264429 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a3b6aaf-4347-4b64-8681-e1218d8f5422-kube-api-access-xtvv6\" (UniqueName: \"kubernetes.io/projected/1a3b6aaf-4347-4b64-8681-e1218d8f5422-kube-api-access-xtvv6\") pod \"1a3b6aaf-4347-4b64-8681-e1218d8f5422\" (UID: \"1a3b6aaf-4347-4b64-8681-e1218d8f5422\") " Oct 30 20:40:35.264814 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:35.264760 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a3b6aaf-4347-4b64-8681-e1218d8f5422-maas-api-openshift-ca" pod "1a3b6aaf-4347-4b64-8681-e1218d8f5422" (UID: "1a3b6aaf-4347-4b64-8681-e1218d8f5422"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:40:35.266545 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:35.266516 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a3b6aaf-4347-4b64-8681-e1218d8f5422-maas-api-xks-ca" pod "1a3b6aaf-4347-4b64-8681-e1218d8f5422" (UID: "1a3b6aaf-4347-4b64-8681-e1218d8f5422"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:40:35.266671 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:35.266560 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a3b6aaf-4347-4b64-8681-e1218d8f5422-kube-api-access-xtvv6" pod "1a3b6aaf-4347-4b64-8681-e1218d8f5422" (UID: "1a3b6aaf-4347-4b64-8681-e1218d8f5422"). InnerVolumeSpecName "kube-api-access-xtvv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:40:35.365082 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:35.364995 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a3b6aaf-4347-4b64-8681-e1218d8f5422-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:40:35.365082 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:35.365023 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a3b6aaf-4347-4b64-8681-e1218d8f5422-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:40:35.365082 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:35.365034 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtvv6\" (UniqueName: \"kubernetes.io/projected/1a3b6aaf-4347-4b64-8681-e1218d8f5422-kube-api-access-xtvv6\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:40:36.050276 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:36.050250 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-273c1b-hwpg5" Oct 30 20:40:36.050459 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:36.050254 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-273c1b-hwpg5" event={"ID":"1a3b6aaf-4347-4b64-8681-e1218d8f5422","Type":"ContainerDied","Data":"7efb8723522a22761882ac83e73dbbcde108752c9f66196b3fa7cb280d49d73d"} Oct 30 20:40:36.050459 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:36.050387 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7efb8723522a22761882ac83e73dbbcde108752c9f66196b3fa7cb280d49d73d" Oct 30 20:40:43.219665 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:40:43.219580 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:40:49.689947 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:49.689905 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-273c1b-787f5dc9d4-pt68w"] Oct 30 20:40:49.690478 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:49.690158 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-273c1b-787f5dc9d4-pt68w" podUID="6f481086-d4fb-403b-822c-bf182959c81f" containerName="maas-api" containerID="cri-o://d60512a22e5a14c8a1d2b383079585a6178fb0b90b6f48bef47c1a2339c16c87" gracePeriod=30 Oct 30 20:40:49.943122 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:49.943059 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-273c1b-787f5dc9d4-pt68w" Oct 30 20:40:49.991382 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:49.991336 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f481086-d4fb-403b-822c-bf182959c81f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f481086-d4fb-403b-822c-bf182959c81f-openshift-service-ca\") pod \"6f481086-d4fb-403b-822c-bf182959c81f\" (UID: \"6f481086-d4fb-403b-822c-bf182959c81f\") " Oct 30 20:40:49.991541 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:49.991397 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f481086-d4fb-403b-822c-bf182959c81f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f481086-d4fb-403b-822c-bf182959c81f-maas-api-tls\") pod \"6f481086-d4fb-403b-822c-bf182959c81f\" (UID: \"6f481086-d4fb-403b-822c-bf182959c81f\") " Oct 30 20:40:49.991541 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:49.991451 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f481086-d4fb-403b-822c-bf182959c81f-kube-api-access-ds7ld\" (UniqueName: \"kubernetes.io/projected/6f481086-d4fb-403b-822c-bf182959c81f-kube-api-access-ds7ld\") pod \"6f481086-d4fb-403b-822c-bf182959c81f\" (UID: \"6f481086-d4fb-403b-822c-bf182959c81f\") " Oct 30 20:40:49.991541 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:49.991505 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f481086-d4fb-403b-822c-bf182959c81f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f481086-d4fb-403b-822c-bf182959c81f-metrics-certs\") pod \"6f481086-d4fb-403b-822c-bf182959c81f\" (UID: \"6f481086-d4fb-403b-822c-bf182959c81f\") " Oct 30 20:40:49.991761 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:49.991734 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f481086-d4fb-403b-822c-bf182959c81f-openshift-service-ca" pod "6f481086-d4fb-403b-822c-bf182959c81f" (UID: "6f481086-d4fb-403b-822c-bf182959c81f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:40:49.993538 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:49.993515 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f481086-d4fb-403b-822c-bf182959c81f-maas-api-tls" pod "6f481086-d4fb-403b-822c-bf182959c81f" (UID: "6f481086-d4fb-403b-822c-bf182959c81f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:40:49.993625 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:49.993556 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f481086-d4fb-403b-822c-bf182959c81f-kube-api-access-ds7ld" pod "6f481086-d4fb-403b-822c-bf182959c81f" (UID: "6f481086-d4fb-403b-822c-bf182959c81f"). InnerVolumeSpecName "kube-api-access-ds7ld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:40:49.993719 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:49.993700 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f481086-d4fb-403b-822c-bf182959c81f-metrics-certs" pod "6f481086-d4fb-403b-822c-bf182959c81f" (UID: "6f481086-d4fb-403b-822c-bf182959c81f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:40:50.093035 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:50.093009 4292 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f481086-d4fb-403b-822c-bf182959c81f-metrics-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:40:50.093035 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:50.093033 4292 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f481086-d4fb-403b-822c-bf182959c81f-openshift-service-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:40:50.093216 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:50.093043 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f481086-d4fb-403b-822c-bf182959c81f-maas-api-tls\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:40:50.093216 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:50.093052 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ds7ld\" (UniqueName: \"kubernetes.io/projected/6f481086-d4fb-403b-822c-bf182959c81f-kube-api-access-ds7ld\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:40:50.120218 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:50.120186 4292 generic.go:356] "Generic (PLEG): container finished" podID="6f481086-d4fb-403b-822c-bf182959c81f" containerID="d60512a22e5a14c8a1d2b383079585a6178fb0b90b6f48bef47c1a2339c16c87" exitCode=0 Oct 30 20:40:50.120364 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:50.120234 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-273c1b-787f5dc9d4-pt68w" Oct 30 20:40:50.120364 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:50.120281 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-273c1b-787f5dc9d4-pt68w" event={"ID":"6f481086-d4fb-403b-822c-bf182959c81f","Type":"ContainerDied","Data":"d60512a22e5a14c8a1d2b383079585a6178fb0b90b6f48bef47c1a2339c16c87"} Oct 30 20:40:50.120364 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:50.120335 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-273c1b-787f5dc9d4-pt68w" event={"ID":"6f481086-d4fb-403b-822c-bf182959c81f","Type":"ContainerDied","Data":"877e373a53f0fb7303be35213fa29c8f1d59dfd4220c6d5ac8277e0b535d2c4c"} Oct 30 20:40:50.120484 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:50.120387 4292 scope.go:122] "RemoveContainer" containerID="d60512a22e5a14c8a1d2b383079585a6178fb0b90b6f48bef47c1a2339c16c87" Oct 30 20:40:50.131701 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:50.131683 4292 scope.go:122] "RemoveContainer" containerID="d60512a22e5a14c8a1d2b383079585a6178fb0b90b6f48bef47c1a2339c16c87" Oct 30 20:40:50.132181 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:40:50.132105 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d60512a22e5a14c8a1d2b383079585a6178fb0b90b6f48bef47c1a2339c16c87\": container with ID starting with d60512a22e5a14c8a1d2b383079585a6178fb0b90b6f48bef47c1a2339c16c87 not found: ID does not exist" containerID="d60512a22e5a14c8a1d2b383079585a6178fb0b90b6f48bef47c1a2339c16c87" Oct 30 20:40:50.132181 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:50.132143 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d60512a22e5a14c8a1d2b383079585a6178fb0b90b6f48bef47c1a2339c16c87"} err="failed to get container status \"d60512a22e5a14c8a1d2b383079585a6178fb0b90b6f48bef47c1a2339c16c87\": rpc error: code = NotFound desc = could not find container \"d60512a22e5a14c8a1d2b383079585a6178fb0b90b6f48bef47c1a2339c16c87\": container with ID starting with d60512a22e5a14c8a1d2b383079585a6178fb0b90b6f48bef47c1a2339c16c87 not found: ID does not exist" Oct 30 20:40:50.288496 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:50.288466 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-273c1b-787f5dc9d4-pt68w"] Oct 30 20:40:50.294183 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:50.294155 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-273c1b-787f5dc9d4-pt68w"] Oct 30 20:40:50.580632 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:50.580561 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f481086-d4fb-403b-822c-bf182959c81f" path="/var/lib/kubelet/pods/6f481086-d4fb-403b-822c-bf182959c81f/volumes" Oct 30 20:40:51.892641 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:51.892601 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-273c1b-hwpg5"] Oct 30 20:40:51.943340 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:51.943305 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-273c1b-hwpg5"] Oct 30 20:40:52.580654 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:52.580626 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a3b6aaf-4347-4b64-8681-e1218d8f5422" path="/var/lib/kubelet/pods/1a3b6aaf-4347-4b64-8681-e1218d8f5422/volumes" Oct 30 20:40:58.493763 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:58.493730 4292 scope.go:122] "RemoveContainer" containerID="1f0248404cc8b4ad9e06ef6e4ce614dc1d6aa871c6b3a0ceaf2dd957bf984a2f" Oct 30 20:40:58.502098 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:58.502076 4292 scope.go:122] "RemoveContainer" containerID="74db238b53b85c63a54657b90aea43b97008b23dd107d72774f4a659ed4bf226" Oct 30 20:40:58.509692 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:58.509673 4292 scope.go:122] "RemoveContainer" containerID="655400de033270da71a67e78a2aeaa16d9b012b11cda17d7609bad74f30305ed" Oct 30 20:40:58.516776 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:40:58.516757 4292 scope.go:122] "RemoveContainer" containerID="f8644de2cd0609206ecb3da5e1afa59900b8fb9413fe896e7becf025b6a442ee" Oct 30 20:41:00.121086 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:00.121053 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-056a27-j5g6b"] Oct 30 20:41:00.121545 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:00.121399 4292 cpu_manager.go:414] "containerMap: removing container" podUID="6f481086-d4fb-403b-822c-bf182959c81f" containerName="maas-api" Oct 30 20:41:00.121545 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:00.121414 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f481086-d4fb-403b-822c-bf182959c81f" containerName="maas-api" Oct 30 20:41:00.121545 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:00.121435 4292 cpu_manager.go:414] "containerMap: removing container" podUID="1a3b6aaf-4347-4b64-8681-e1218d8f5422" containerName="revoke-keys" Oct 30 20:41:00.121545 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:00.121441 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a3b6aaf-4347-4b64-8681-e1218d8f5422" containerName="revoke-keys" Oct 30 20:41:00.121545 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:00.121511 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f481086-d4fb-403b-822c-bf182959c81f" containerName="maas-api" Oct 30 20:41:00.121545 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:00.121520 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a3b6aaf-4347-4b64-8681-e1218d8f5422" containerName="revoke-keys" Oct 30 20:41:00.126339 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:00.126319 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-056a27-j5g6b" Oct 30 20:41:00.191546 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:00.191508 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-056a27-j5g6b"] Oct 30 20:41:00.229752 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:00.229727 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:41:00.229752 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:00.229727 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:41:00.229752 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:00.229729 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:41:00.233428 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:00.233413 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\"" Oct 30 20:41:00.289827 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:00.289803 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wl9k\" (UniqueName: \"kubernetes.io/projected/d60e9db0-8b78-4586-9f7f-91c7985dd1ae-kube-api-access-2wl9k\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-056a27-j5g6b\" (UID: \"d60e9db0-8b78-4586-9f7f-91c7985dd1ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-056a27-j5g6b" Oct 30 20:41:00.289988 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:00.289860 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d60e9db0-8b78-4586-9f7f-91c7985dd1ae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-056a27-j5g6b\" (UID: \"d60e9db0-8b78-4586-9f7f-91c7985dd1ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-056a27-j5g6b" Oct 30 20:41:00.289988 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:00.289959 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d60e9db0-8b78-4586-9f7f-91c7985dd1ae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-056a27-j5g6b\" (UID: \"d60e9db0-8b78-4586-9f7f-91c7985dd1ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-056a27-j5g6b" Oct 30 20:41:00.390892 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:00.390825 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d60e9db0-8b78-4586-9f7f-91c7985dd1ae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-056a27-j5g6b\" (UID: \"d60e9db0-8b78-4586-9f7f-91c7985dd1ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-056a27-j5g6b" Oct 30 20:41:00.390892 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:00.390873 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d60e9db0-8b78-4586-9f7f-91c7985dd1ae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-056a27-j5g6b\" (UID: \"d60e9db0-8b78-4586-9f7f-91c7985dd1ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-056a27-j5g6b" Oct 30 20:41:00.391107 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:00.390898 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wl9k\" (UniqueName: \"kubernetes.io/projected/d60e9db0-8b78-4586-9f7f-91c7985dd1ae-kube-api-access-2wl9k\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-056a27-j5g6b\" (UID: \"d60e9db0-8b78-4586-9f7f-91c7985dd1ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-056a27-j5g6b" Oct 30 20:41:00.391475 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:00.391455 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d60e9db0-8b78-4586-9f7f-91c7985dd1ae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-056a27-j5g6b\" (UID: \"d60e9db0-8b78-4586-9f7f-91c7985dd1ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-056a27-j5g6b" Oct 30 20:41:00.393232 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:00.393210 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d60e9db0-8b78-4586-9f7f-91c7985dd1ae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-056a27-j5g6b\" (UID: \"d60e9db0-8b78-4586-9f7f-91c7985dd1ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-056a27-j5g6b" Oct 30 20:41:00.453556 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:00.453534 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wl9k\" (UniqueName: \"kubernetes.io/projected/d60e9db0-8b78-4586-9f7f-91c7985dd1ae-kube-api-access-2wl9k\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-056a27-j5g6b\" (UID: \"d60e9db0-8b78-4586-9f7f-91c7985dd1ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-056a27-j5g6b" Oct 30 20:41:00.738843 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:00.738770 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-056a27-j5g6b" Oct 30 20:41:01.112166 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:01.112139 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-056a27-j5g6b"] Oct 30 20:41:01.114193 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:41:01.114159 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd60e9db0_8b78_4586_9f7f_91c7985dd1ae.slice/crio-a76b40ea1b35efc2efc9b5f3804c82d7a997e5ce57c30027e56fe9e07f7d1261 WatchSource:0}: Error finding container a76b40ea1b35efc2efc9b5f3804c82d7a997e5ce57c30027e56fe9e07f7d1261: Status 404 returned error can't find the container with id a76b40ea1b35efc2efc9b5f3804c82d7a997e5ce57c30027e56fe9e07f7d1261 Oct 30 20:41:01.185090 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:01.185058 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-056a27-j5g6b" event={"ID":"d60e9db0-8b78-4586-9f7f-91c7985dd1ae","Type":"ContainerStarted","Data":"a76b40ea1b35efc2efc9b5f3804c82d7a997e5ce57c30027e56fe9e07f7d1261"} Oct 30 20:41:02.192976 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:02.192890 4292 generic.go:356] "Generic (PLEG): container finished" podID="d60e9db0-8b78-4586-9f7f-91c7985dd1ae" containerID="68239e9c03b277689520f9e4fc6392d5a4cb3f27c65f38fef0caffcab7d7f72d" exitCode=0 Oct 30 20:41:02.193395 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:02.192984 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-056a27-j5g6b" event={"ID":"d60e9db0-8b78-4586-9f7f-91c7985dd1ae","Type":"ContainerDied","Data":"68239e9c03b277689520f9e4fc6392d5a4cb3f27c65f38fef0caffcab7d7f72d"} Oct 30 20:41:03.319548 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:03.319522 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-056a27-j5g6b" Oct 30 20:41:03.417421 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:03.417371 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d60e9db0-8b78-4586-9f7f-91c7985dd1ae-kube-api-access-2wl9k\" (UniqueName: \"kubernetes.io/projected/d60e9db0-8b78-4586-9f7f-91c7985dd1ae-kube-api-access-2wl9k\") pod \"d60e9db0-8b78-4586-9f7f-91c7985dd1ae\" (UID: \"d60e9db0-8b78-4586-9f7f-91c7985dd1ae\") " Oct 30 20:41:03.417600 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:03.417445 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d60e9db0-8b78-4586-9f7f-91c7985dd1ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d60e9db0-8b78-4586-9f7f-91c7985dd1ae-maas-api-xks-ca\") pod \"d60e9db0-8b78-4586-9f7f-91c7985dd1ae\" (UID: \"d60e9db0-8b78-4586-9f7f-91c7985dd1ae\") " Oct 30 20:41:03.417600 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:03.417545 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d60e9db0-8b78-4586-9f7f-91c7985dd1ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d60e9db0-8b78-4586-9f7f-91c7985dd1ae-maas-api-openshift-ca\") pod \"d60e9db0-8b78-4586-9f7f-91c7985dd1ae\" (UID: \"d60e9db0-8b78-4586-9f7f-91c7985dd1ae\") " Oct 30 20:41:03.417885 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:03.417864 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d60e9db0-8b78-4586-9f7f-91c7985dd1ae-maas-api-openshift-ca" pod "d60e9db0-8b78-4586-9f7f-91c7985dd1ae" (UID: "d60e9db0-8b78-4586-9f7f-91c7985dd1ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:41:03.419445 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:03.419424 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d60e9db0-8b78-4586-9f7f-91c7985dd1ae-maas-api-xks-ca" pod "d60e9db0-8b78-4586-9f7f-91c7985dd1ae" (UID: "d60e9db0-8b78-4586-9f7f-91c7985dd1ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:41:03.419525 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:03.419499 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d60e9db0-8b78-4586-9f7f-91c7985dd1ae-kube-api-access-2wl9k" pod "d60e9db0-8b78-4586-9f7f-91c7985dd1ae" (UID: "d60e9db0-8b78-4586-9f7f-91c7985dd1ae"). InnerVolumeSpecName "kube-api-access-2wl9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:41:03.518646 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:03.518621 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d60e9db0-8b78-4586-9f7f-91c7985dd1ae-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:41:03.518646 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:03.518647 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d60e9db0-8b78-4586-9f7f-91c7985dd1ae-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:41:03.518833 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:03.518656 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wl9k\" (UniqueName: \"kubernetes.io/projected/d60e9db0-8b78-4586-9f7f-91c7985dd1ae-kube-api-access-2wl9k\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:41:04.203113 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:04.203076 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-056a27-j5g6b" event={"ID":"d60e9db0-8b78-4586-9f7f-91c7985dd1ae","Type":"ContainerDied","Data":"a76b40ea1b35efc2efc9b5f3804c82d7a997e5ce57c30027e56fe9e07f7d1261"} Oct 30 20:41:04.203113 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:04.203099 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-056a27-j5g6b" Oct 30 20:41:04.203343 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:04.203136 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a76b40ea1b35efc2efc9b5f3804c82d7a997e5ce57c30027e56fe9e07f7d1261" Oct 30 20:41:13.219702 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:41:13.219665 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:41:24.078264 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:24.078223 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-056a27-j5g6b"] Oct 30 20:41:24.128834 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:24.128797 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-056a27-j5g6b"] Oct 30 20:41:24.580950 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:24.580917 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d60e9db0-8b78-4586-9f7f-91c7985dd1ae" path="/var/lib/kubelet/pods/d60e9db0-8b78-4586-9f7f-91c7985dd1ae/volumes" Oct 30 20:41:31.793843 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:31.793812 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh"] Oct 30 20:41:31.794387 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:31.794033 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh" podUID="ec42d060-8ef1-46d2-9bb8-4ebb4224389b" containerName="istio-proxy" containerID="cri-o://56ca070e9b9b4b4d7e16e0a030304d51db75323d387f17b6af1d515efbee4550" gracePeriod=30 Oct 30 20:41:32.711708 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:32.711674 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-64120b-zkcwm"] Oct 30 20:41:32.712002 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:32.711990 4292 cpu_manager.go:414] "containerMap: removing container" podUID="d60e9db0-8b78-4586-9f7f-91c7985dd1ae" containerName="revoke-keys" Oct 30 20:41:32.712050 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:32.712003 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d60e9db0-8b78-4586-9f7f-91c7985dd1ae" containerName="revoke-keys" Oct 30 20:41:32.712088 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:32.712067 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="d60e9db0-8b78-4586-9f7f-91c7985dd1ae" containerName="revoke-keys" Oct 30 20:41:32.715106 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:32.715091 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-64120b-zkcwm" Oct 30 20:41:32.750587 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:32.750560 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e97a61c0-0eb9-4d91-a5c3-c45ad9b16425-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-64120b-zkcwm\" (UID: \"e97a61c0-0eb9-4d91-a5c3-c45ad9b16425\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-64120b-zkcwm" Oct 30 20:41:32.750724 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:32.750634 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e97a61c0-0eb9-4d91-a5c3-c45ad9b16425-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-64120b-zkcwm\" (UID: \"e97a61c0-0eb9-4d91-a5c3-c45ad9b16425\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-64120b-zkcwm" Oct 30 20:41:32.750724 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:32.750671 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg7zc\" (UniqueName: \"kubernetes.io/projected/e97a61c0-0eb9-4d91-a5c3-c45ad9b16425-kube-api-access-qg7zc\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-64120b-zkcwm\" (UID: \"e97a61c0-0eb9-4d91-a5c3-c45ad9b16425\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-64120b-zkcwm" Oct 30 20:41:32.764921 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:32.764894 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-64120b-zkcwm"] Oct 30 20:41:32.769266 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:32.769241 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:41:32.769428 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:32.769391 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:41:32.814112 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:32.814077 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\"" Oct 30 20:41:32.814112 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:32.814094 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:41:32.851431 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:32.851343 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e97a61c0-0eb9-4d91-a5c3-c45ad9b16425-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-64120b-zkcwm\" (UID: \"e97a61c0-0eb9-4d91-a5c3-c45ad9b16425\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-64120b-zkcwm" Oct 30 20:41:32.851564 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:32.851507 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e97a61c0-0eb9-4d91-a5c3-c45ad9b16425-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-64120b-zkcwm\" (UID: \"e97a61c0-0eb9-4d91-a5c3-c45ad9b16425\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-64120b-zkcwm" Oct 30 20:41:32.851564 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:32.851531 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg7zc\" (UniqueName: \"kubernetes.io/projected/e97a61c0-0eb9-4d91-a5c3-c45ad9b16425-kube-api-access-qg7zc\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-64120b-zkcwm\" (UID: \"e97a61c0-0eb9-4d91-a5c3-c45ad9b16425\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-64120b-zkcwm" Oct 30 20:41:32.851966 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:32.851944 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e97a61c0-0eb9-4d91-a5c3-c45ad9b16425-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-64120b-zkcwm\" (UID: \"e97a61c0-0eb9-4d91-a5c3-c45ad9b16425\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-64120b-zkcwm" Oct 30 20:41:32.853815 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:32.853794 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e97a61c0-0eb9-4d91-a5c3-c45ad9b16425-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-64120b-zkcwm\" (UID: \"e97a61c0-0eb9-4d91-a5c3-c45ad9b16425\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-64120b-zkcwm" Oct 30 20:41:32.926215 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:32.926185 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg7zc\" (UniqueName: \"kubernetes.io/projected/e97a61c0-0eb9-4d91-a5c3-c45ad9b16425-kube-api-access-qg7zc\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-64120b-zkcwm\" (UID: \"e97a61c0-0eb9-4d91-a5c3-c45ad9b16425\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-64120b-zkcwm" Oct 30 20:41:33.027039 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:33.027016 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-64120b-zkcwm" Oct 30 20:41:33.409013 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:33.408984 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-64120b-zkcwm"] Oct 30 20:41:33.411130 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:41:33.411101 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode97a61c0_0eb9_4d91_a5c3_c45ad9b16425.slice/crio-8514fa23538a6d650af937b7a9bcabdaedc7f6d869f199f214482f35f06573d2 WatchSource:0}: Error finding container 8514fa23538a6d650af937b7a9bcabdaedc7f6d869f199f214482f35f06573d2: Status 404 returned error can't find the container with id 8514fa23538a6d650af937b7a9bcabdaedc7f6d869f199f214482f35f06573d2 Oct 30 20:41:34.345234 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:34.345198 4292 generic.go:356] "Generic (PLEG): container finished" podID="e97a61c0-0eb9-4d91-a5c3-c45ad9b16425" containerID="1d4ddd32721adf07fc4c6d2757014e2166b481501615ae88e35cbd7d19aa0134" exitCode=0 Oct 30 20:41:34.345682 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:34.345283 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-64120b-zkcwm" event={"ID":"e97a61c0-0eb9-4d91-a5c3-c45ad9b16425","Type":"ContainerDied","Data":"1d4ddd32721adf07fc4c6d2757014e2166b481501615ae88e35cbd7d19aa0134"} Oct 30 20:41:34.345682 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:34.345343 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-64120b-zkcwm" event={"ID":"e97a61c0-0eb9-4d91-a5c3-c45ad9b16425","Type":"ContainerStarted","Data":"8514fa23538a6d650af937b7a9bcabdaedc7f6d869f199f214482f35f06573d2"} Oct 30 20:41:35.475640 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:35.475606 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-64120b-zkcwm" Oct 30 20:41:35.571465 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:35.571431 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e97a61c0-0eb9-4d91-a5c3-c45ad9b16425-kube-api-access-qg7zc\" (UniqueName: \"kubernetes.io/projected/e97a61c0-0eb9-4d91-a5c3-c45ad9b16425-kube-api-access-qg7zc\") pod \"e97a61c0-0eb9-4d91-a5c3-c45ad9b16425\" (UID: \"e97a61c0-0eb9-4d91-a5c3-c45ad9b16425\") " Oct 30 20:41:35.571631 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:35.571475 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e97a61c0-0eb9-4d91-a5c3-c45ad9b16425-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e97a61c0-0eb9-4d91-a5c3-c45ad9b16425-maas-api-openshift-ca\") pod \"e97a61c0-0eb9-4d91-a5c3-c45ad9b16425\" (UID: \"e97a61c0-0eb9-4d91-a5c3-c45ad9b16425\") " Oct 30 20:41:35.571631 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:35.571580 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e97a61c0-0eb9-4d91-a5c3-c45ad9b16425-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e97a61c0-0eb9-4d91-a5c3-c45ad9b16425-maas-api-xks-ca\") pod \"e97a61c0-0eb9-4d91-a5c3-c45ad9b16425\" (UID: \"e97a61c0-0eb9-4d91-a5c3-c45ad9b16425\") " Oct 30 20:41:35.571904 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:35.571876 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e97a61c0-0eb9-4d91-a5c3-c45ad9b16425-maas-api-openshift-ca" pod "e97a61c0-0eb9-4d91-a5c3-c45ad9b16425" (UID: "e97a61c0-0eb9-4d91-a5c3-c45ad9b16425"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:41:35.573603 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:35.573576 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e97a61c0-0eb9-4d91-a5c3-c45ad9b16425-maas-api-xks-ca" pod "e97a61c0-0eb9-4d91-a5c3-c45ad9b16425" (UID: "e97a61c0-0eb9-4d91-a5c3-c45ad9b16425"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:41:35.573731 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:35.573628 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e97a61c0-0eb9-4d91-a5c3-c45ad9b16425-kube-api-access-qg7zc" pod "e97a61c0-0eb9-4d91-a5c3-c45ad9b16425" (UID: "e97a61c0-0eb9-4d91-a5c3-c45ad9b16425"). InnerVolumeSpecName "kube-api-access-qg7zc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:41:35.672980 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:35.672909 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e97a61c0-0eb9-4d91-a5c3-c45ad9b16425-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:41:35.672980 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:35.672934 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qg7zc\" (UniqueName: \"kubernetes.io/projected/e97a61c0-0eb9-4d91-a5c3-c45ad9b16425-kube-api-access-qg7zc\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:41:35.672980 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:35.672946 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e97a61c0-0eb9-4d91-a5c3-c45ad9b16425-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:41:36.246841 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:36.246807 4292 patch_prober.go:29] interesting pod/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:41:36.247011 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:36.246862 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh" podUID="ec42d060-8ef1-46d2-9bb8-4ebb4224389b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:41:36.357318 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:36.357287 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-64120b-zkcwm" Oct 30 20:41:36.357536 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:36.357332 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-64120b-zkcwm" event={"ID":"e97a61c0-0eb9-4d91-a5c3-c45ad9b16425","Type":"ContainerDied","Data":"8514fa23538a6d650af937b7a9bcabdaedc7f6d869f199f214482f35f06573d2"} Oct 30 20:41:36.357536 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:36.357426 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8514fa23538a6d650af937b7a9bcabdaedc7f6d869f199f214482f35f06573d2" Oct 30 20:41:37.031892 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:37.031872 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh" Oct 30 20:41:37.084392 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:37.084301 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec42d060-8ef1-46d2-9bb8-4ebb4224389b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec42d060-8ef1-46d2-9bb8-4ebb4224389b-workload-certs\") pod \"ec42d060-8ef1-46d2-9bb8-4ebb4224389b\" (UID: \"ec42d060-8ef1-46d2-9bb8-4ebb4224389b\") " Oct 30 20:41:37.084392 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:37.084367 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec42d060-8ef1-46d2-9bb8-4ebb4224389b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec42d060-8ef1-46d2-9bb8-4ebb4224389b-istio-envoy\") pod \"ec42d060-8ef1-46d2-9bb8-4ebb4224389b\" (UID: \"ec42d060-8ef1-46d2-9bb8-4ebb4224389b\") " Oct 30 20:41:37.084636 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:37.084435 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ec42d060-8ef1-46d2-9bb8-4ebb4224389b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec42d060-8ef1-46d2-9bb8-4ebb4224389b-istio-podinfo\") pod \"ec42d060-8ef1-46d2-9bb8-4ebb4224389b\" (UID: \"ec42d060-8ef1-46d2-9bb8-4ebb4224389b\") " Oct 30 20:41:37.084636 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:37.084524 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec42d060-8ef1-46d2-9bb8-4ebb4224389b-kube-api-access-gvhx8\" (UniqueName: \"kubernetes.io/projected/ec42d060-8ef1-46d2-9bb8-4ebb4224389b-kube-api-access-gvhx8\") pod \"ec42d060-8ef1-46d2-9bb8-4ebb4224389b\" (UID: \"ec42d060-8ef1-46d2-9bb8-4ebb4224389b\") " Oct 30 20:41:37.084636 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:37.084560 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec42d060-8ef1-46d2-9bb8-4ebb4224389b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec42d060-8ef1-46d2-9bb8-4ebb4224389b-istiod-ca-cert\") pod \"ec42d060-8ef1-46d2-9bb8-4ebb4224389b\" (UID: \"ec42d060-8ef1-46d2-9bb8-4ebb4224389b\") " Oct 30 20:41:37.084636 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:37.084563 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec42d060-8ef1-46d2-9bb8-4ebb4224389b-workload-certs" pod "ec42d060-8ef1-46d2-9bb8-4ebb4224389b" (UID: "ec42d060-8ef1-46d2-9bb8-4ebb4224389b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:41:37.084636 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:37.084599 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec42d060-8ef1-46d2-9bb8-4ebb4224389b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec42d060-8ef1-46d2-9bb8-4ebb4224389b-workload-socket\") pod \"ec42d060-8ef1-46d2-9bb8-4ebb4224389b\" (UID: \"ec42d060-8ef1-46d2-9bb8-4ebb4224389b\") " Oct 30 20:41:37.084915 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:37.084644 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec42d060-8ef1-46d2-9bb8-4ebb4224389b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec42d060-8ef1-46d2-9bb8-4ebb4224389b-istio-data\") pod \"ec42d060-8ef1-46d2-9bb8-4ebb4224389b\" (UID: \"ec42d060-8ef1-46d2-9bb8-4ebb4224389b\") " Oct 30 20:41:37.084915 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:37.084684 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec42d060-8ef1-46d2-9bb8-4ebb4224389b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec42d060-8ef1-46d2-9bb8-4ebb4224389b-credential-socket\") pod \"ec42d060-8ef1-46d2-9bb8-4ebb4224389b\" (UID: \"ec42d060-8ef1-46d2-9bb8-4ebb4224389b\") " Oct 30 20:41:37.084915 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:37.084716 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec42d060-8ef1-46d2-9bb8-4ebb4224389b-istio-token\" (UniqueName: \"kubernetes.io/projected/ec42d060-8ef1-46d2-9bb8-4ebb4224389b-istio-token\") pod \"ec42d060-8ef1-46d2-9bb8-4ebb4224389b\" (UID: \"ec42d060-8ef1-46d2-9bb8-4ebb4224389b\") " Oct 30 20:41:37.084915 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:37.084794 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec42d060-8ef1-46d2-9bb8-4ebb4224389b-workload-socket" pod "ec42d060-8ef1-46d2-9bb8-4ebb4224389b" (UID: "ec42d060-8ef1-46d2-9bb8-4ebb4224389b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:41:37.085120 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:37.084960 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec42d060-8ef1-46d2-9bb8-4ebb4224389b-credential-socket" pod "ec42d060-8ef1-46d2-9bb8-4ebb4224389b" (UID: "ec42d060-8ef1-46d2-9bb8-4ebb4224389b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:41:37.085120 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:37.084979 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec42d060-8ef1-46d2-9bb8-4ebb4224389b-istio-data" pod "ec42d060-8ef1-46d2-9bb8-4ebb4224389b" (UID: "ec42d060-8ef1-46d2-9bb8-4ebb4224389b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:41:37.085120 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:37.085036 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec42d060-8ef1-46d2-9bb8-4ebb4224389b-workload-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:41:37.085120 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:37.085057 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec42d060-8ef1-46d2-9bb8-4ebb4224389b-workload-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:41:37.085120 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:37.085106 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec42d060-8ef1-46d2-9bb8-4ebb4224389b-istiod-ca-cert" pod "ec42d060-8ef1-46d2-9bb8-4ebb4224389b" (UID: "ec42d060-8ef1-46d2-9bb8-4ebb4224389b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:41:37.086902 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:37.086877 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec42d060-8ef1-46d2-9bb8-4ebb4224389b-istio-podinfo" pod "ec42d060-8ef1-46d2-9bb8-4ebb4224389b" (UID: "ec42d060-8ef1-46d2-9bb8-4ebb4224389b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:41:37.087021 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:37.086953 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec42d060-8ef1-46d2-9bb8-4ebb4224389b-istio-token" pod "ec42d060-8ef1-46d2-9bb8-4ebb4224389b" (UID: "ec42d060-8ef1-46d2-9bb8-4ebb4224389b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:41:37.087182 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:37.087163 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec42d060-8ef1-46d2-9bb8-4ebb4224389b-istio-envoy" pod "ec42d060-8ef1-46d2-9bb8-4ebb4224389b" (UID: "ec42d060-8ef1-46d2-9bb8-4ebb4224389b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:41:37.087235 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:37.087174 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec42d060-8ef1-46d2-9bb8-4ebb4224389b-kube-api-access-gvhx8" pod "ec42d060-8ef1-46d2-9bb8-4ebb4224389b" (UID: "ec42d060-8ef1-46d2-9bb8-4ebb4224389b"). InnerVolumeSpecName "kube-api-access-gvhx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:41:37.186296 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:37.186259 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec42d060-8ef1-46d2-9bb8-4ebb4224389b-credential-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:41:37.186296 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:37.186289 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec42d060-8ef1-46d2-9bb8-4ebb4224389b-istio-token\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:41:37.186296 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:37.186298 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec42d060-8ef1-46d2-9bb8-4ebb4224389b-istio-envoy\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:41:37.186615 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:37.186306 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec42d060-8ef1-46d2-9bb8-4ebb4224389b-istio-podinfo\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:41:37.186615 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:37.186315 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvhx8\" (UniqueName: \"kubernetes.io/projected/ec42d060-8ef1-46d2-9bb8-4ebb4224389b-kube-api-access-gvhx8\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:41:37.186615 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:37.186324 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec42d060-8ef1-46d2-9bb8-4ebb4224389b-istiod-ca-cert\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:41:37.186615 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:37.186332 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec42d060-8ef1-46d2-9bb8-4ebb4224389b-istio-data\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:41:37.364452 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:37.364335 4292 generic.go:356] "Generic (PLEG): container finished" podID="ec42d060-8ef1-46d2-9bb8-4ebb4224389b" containerID="56ca070e9b9b4b4d7e16e0a030304d51db75323d387f17b6af1d515efbee4550" exitCode=0 Oct 30 20:41:37.364452 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:37.364408 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh" Oct 30 20:41:37.364452 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:37.364422 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh" event={"ID":"ec42d060-8ef1-46d2-9bb8-4ebb4224389b","Type":"ContainerDied","Data":"56ca070e9b9b4b4d7e16e0a030304d51db75323d387f17b6af1d515efbee4550"} Oct 30 20:41:37.364750 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:37.364483 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh" event={"ID":"ec42d060-8ef1-46d2-9bb8-4ebb4224389b","Type":"ContainerDied","Data":"b1b7406f5cdd55f97301cdf56dff12b7f6ad72029c4776fdea8a6187226452f7"} Oct 30 20:41:37.364750 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:37.364512 4292 scope.go:122] "RemoveContainer" containerID="56ca070e9b9b4b4d7e16e0a030304d51db75323d387f17b6af1d515efbee4550" Oct 30 20:41:37.374095 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:37.374074 4292 scope.go:122] "RemoveContainer" containerID="fa7e62302e8ab4f89ed8e1c71a650a09ad421e825e4239ec7e711a8dbe778d92" Oct 30 20:41:37.382445 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:37.382420 4292 scope.go:122] "RemoveContainer" containerID="56ca070e9b9b4b4d7e16e0a030304d51db75323d387f17b6af1d515efbee4550" Oct 30 20:41:37.382709 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:41:37.382690 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56ca070e9b9b4b4d7e16e0a030304d51db75323d387f17b6af1d515efbee4550\": container with ID starting with 56ca070e9b9b4b4d7e16e0a030304d51db75323d387f17b6af1d515efbee4550 not found: ID does not exist" containerID="56ca070e9b9b4b4d7e16e0a030304d51db75323d387f17b6af1d515efbee4550" Oct 30 20:41:37.382755 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:37.382718 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56ca070e9b9b4b4d7e16e0a030304d51db75323d387f17b6af1d515efbee4550"} err="failed to get container status \"56ca070e9b9b4b4d7e16e0a030304d51db75323d387f17b6af1d515efbee4550\": rpc error: code = NotFound desc = could not find container \"56ca070e9b9b4b4d7e16e0a030304d51db75323d387f17b6af1d515efbee4550\": container with ID starting with 56ca070e9b9b4b4d7e16e0a030304d51db75323d387f17b6af1d515efbee4550 not found: ID does not exist" Oct 30 20:41:37.382755 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:37.382736 4292 scope.go:122] "RemoveContainer" containerID="fa7e62302e8ab4f89ed8e1c71a650a09ad421e825e4239ec7e711a8dbe778d92" Oct 30 20:41:37.382939 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:41:37.382924 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa7e62302e8ab4f89ed8e1c71a650a09ad421e825e4239ec7e711a8dbe778d92\": container with ID starting with fa7e62302e8ab4f89ed8e1c71a650a09ad421e825e4239ec7e711a8dbe778d92 not found: ID does not exist" containerID="fa7e62302e8ab4f89ed8e1c71a650a09ad421e825e4239ec7e711a8dbe778d92" Oct 30 20:41:37.382984 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:37.382946 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa7e62302e8ab4f89ed8e1c71a650a09ad421e825e4239ec7e711a8dbe778d92"} err="failed to get container status \"fa7e62302e8ab4f89ed8e1c71a650a09ad421e825e4239ec7e711a8dbe778d92\": rpc error: code = NotFound desc = could not find container \"fa7e62302e8ab4f89ed8e1c71a650a09ad421e825e4239ec7e711a8dbe778d92\": container with ID starting with fa7e62302e8ab4f89ed8e1c71a650a09ad421e825e4239ec7e711a8dbe778d92 not found: ID does not exist" Oct 30 20:41:37.533091 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:37.533054 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh"] Oct 30 20:41:37.539049 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:37.539021 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-056a27-openshift-default-79ff5c986f-fblgh"] Oct 30 20:41:38.580808 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:38.580777 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec42d060-8ef1-46d2-9bb8-4ebb4224389b" path="/var/lib/kubelet/pods/ec42d060-8ef1-46d2-9bb8-4ebb4224389b/volumes" Oct 30 20:41:43.219447 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:41:43.219406 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:41:47.071860 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:47.071826 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a65f438e-ctj4f"] Oct 30 20:41:47.122323 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:47.122291 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a65f438e-ctj4f"] Oct 30 20:41:48.586262 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:48.586229 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17ecf369-9fcc-4bce-a605-2bde479d0bc6" path="/var/lib/kubelet/pods/17ecf369-9fcc-4bce-a605-2bde479d0bc6/volumes" Oct 30 20:41:54.050323 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:54.050288 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b6ecbe3-hw7jt"] Oct 30 20:41:54.101345 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:54.101298 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce7c24b1-z7zcg"] Oct 30 20:41:54.102826 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:54.102815 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b6ecbe3-hw7jt"] Oct 30 20:41:54.104094 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:54.104074 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce7c24b1-z7zcg"] Oct 30 20:41:54.582653 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:54.582624 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7af30adc-8334-4ea8-bc21-e61f98011790" path="/var/lib/kubelet/pods/7af30adc-8334-4ea8-bc21-e61f98011790/volumes" Oct 30 20:41:54.583045 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:54.583031 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="981830dd-2c55-4876-be0a-84eb1dfb69ae" path="/var/lib/kubelet/pods/981830dd-2c55-4876-be0a-84eb1dfb69ae/volumes" Oct 30 20:41:56.643749 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:56.643716 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-64120b-zkcwm"] Oct 30 20:41:56.694736 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:56.694702 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-64120b-zkcwm"] Oct 30 20:41:58.549001 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:58.548974 4292 scope.go:122] "RemoveContainer" containerID="a7abcd3032dc4ca6a79406215d50ed52da36c1e59451d1651f2d272e5f12efa5" Oct 30 20:41:58.557180 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:58.557160 4292 scope.go:122] "RemoveContainer" containerID="8f57066d64f8a5caa0c562589922a9a98ea16a1638188e7fa831005c85d8a164" Oct 30 20:41:58.581556 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:41:58.581530 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e97a61c0-0eb9-4d91-a5c3-c45ad9b16425" path="/var/lib/kubelet/pods/e97a61c0-0eb9-4d91-a5c3-c45ad9b16425/volumes" Oct 30 20:42:03.753546 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:03.753512 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk"] Oct 30 20:42:03.754034 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:03.753734 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk" podUID="05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f" containerName="istio-proxy" containerID="cri-o://fefa5b1bed1324ee25a7037a3d40050b2cc56acdb453ca4017422905c1a59191" gracePeriod=30 Oct 30 20:42:04.823173 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:04.823138 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b0a183-k26fm"] Oct 30 20:42:04.823578 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:04.823457 4292 cpu_manager.go:414] "containerMap: removing container" podUID="ec42d060-8ef1-46d2-9bb8-4ebb4224389b" containerName="istio-proxy" Oct 30 20:42:04.823578 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:04.823467 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec42d060-8ef1-46d2-9bb8-4ebb4224389b" containerName="istio-proxy" Oct 30 20:42:04.823578 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:04.823507 4292 cpu_manager.go:414] "containerMap: removing container" podUID="ec42d060-8ef1-46d2-9bb8-4ebb4224389b" containerName="istio-proxy" Oct 30 20:42:04.823578 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:04.823513 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec42d060-8ef1-46d2-9bb8-4ebb4224389b" containerName="istio-proxy" Oct 30 20:42:04.823578 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:04.823525 4292 cpu_manager.go:414] "containerMap: removing container" podUID="e97a61c0-0eb9-4d91-a5c3-c45ad9b16425" containerName="revoke-keys" Oct 30 20:42:04.823578 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:04.823531 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e97a61c0-0eb9-4d91-a5c3-c45ad9b16425" containerName="revoke-keys" Oct 30 20:42:04.823578 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:04.823580 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec42d060-8ef1-46d2-9bb8-4ebb4224389b" containerName="istio-proxy" Oct 30 20:42:04.823804 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:04.823588 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="e97a61c0-0eb9-4d91-a5c3-c45ad9b16425" containerName="revoke-keys" Oct 30 20:42:04.823804 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:04.823595 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec42d060-8ef1-46d2-9bb8-4ebb4224389b" containerName="istio-proxy" Oct 30 20:42:04.828192 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:04.828174 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b0a183-k26fm" Oct 30 20:42:04.880871 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:04.880837 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b0a183-k26fm"] Oct 30 20:42:04.925987 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:04.925960 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq72v\" (UniqueName: \"kubernetes.io/projected/47850ba0-2841-4253-a00e-12787f69297f-kube-api-access-nq72v\") pod \"maas-api-revoke-keys-e2e-worker-w2-b0a183-k26fm\" (UID: \"47850ba0-2841-4253-a00e-12787f69297f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b0a183-k26fm" Oct 30 20:42:04.926150 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:04.926012 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47850ba0-2841-4253-a00e-12787f69297f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-b0a183-k26fm\" (UID: \"47850ba0-2841-4253-a00e-12787f69297f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b0a183-k26fm" Oct 30 20:42:04.926150 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:04.926096 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47850ba0-2841-4253-a00e-12787f69297f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-b0a183-k26fm\" (UID: \"47850ba0-2841-4253-a00e-12787f69297f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b0a183-k26fm" Oct 30 20:42:04.929596 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:04.929568 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:42:04.929596 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:04.929583 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:42:04.929776 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:04.929607 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:42:04.932292 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:04.932273 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\"" Oct 30 20:42:05.027149 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:05.027122 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47850ba0-2841-4253-a00e-12787f69297f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-b0a183-k26fm\" (UID: \"47850ba0-2841-4253-a00e-12787f69297f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b0a183-k26fm" Oct 30 20:42:05.027320 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:05.027167 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47850ba0-2841-4253-a00e-12787f69297f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-b0a183-k26fm\" (UID: \"47850ba0-2841-4253-a00e-12787f69297f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b0a183-k26fm" Oct 30 20:42:05.027320 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:05.027216 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq72v\" (UniqueName: \"kubernetes.io/projected/47850ba0-2841-4253-a00e-12787f69297f-kube-api-access-nq72v\") pod \"maas-api-revoke-keys-e2e-worker-w2-b0a183-k26fm\" (UID: \"47850ba0-2841-4253-a00e-12787f69297f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b0a183-k26fm" Oct 30 20:42:05.027796 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:05.027774 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47850ba0-2841-4253-a00e-12787f69297f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-b0a183-k26fm\" (UID: \"47850ba0-2841-4253-a00e-12787f69297f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b0a183-k26fm" Oct 30 20:42:05.029543 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:05.029521 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47850ba0-2841-4253-a00e-12787f69297f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-b0a183-k26fm\" (UID: \"47850ba0-2841-4253-a00e-12787f69297f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b0a183-k26fm" Oct 30 20:42:05.051302 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:05.051276 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a659f45-f2vdb"] Oct 30 20:42:05.054735 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:05.054715 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a659f45-f2vdb"] Oct 30 20:42:05.090211 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:05.090154 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq72v\" (UniqueName: \"kubernetes.io/projected/47850ba0-2841-4253-a00e-12787f69297f-kube-api-access-nq72v\") pod \"maas-api-revoke-keys-e2e-worker-w2-b0a183-k26fm\" (UID: \"47850ba0-2841-4253-a00e-12787f69297f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b0a183-k26fm" Oct 30 20:42:05.141165 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:05.141130 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b0a183-k26fm" Oct 30 20:42:05.513236 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:05.513213 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b0a183-k26fm"] Oct 30 20:42:05.514824 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:42:05.514800 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47850ba0_2841_4253_a00e_12787f69297f.slice/crio-e85b425d14c6f4e6bd73b5a04dc538e1a1d7520fe81c66bb7377f528bdb8ddcd WatchSource:0}: Error finding container e85b425d14c6f4e6bd73b5a04dc538e1a1d7520fe81c66bb7377f528bdb8ddcd: Status 404 returned error can't find the container with id e85b425d14c6f4e6bd73b5a04dc538e1a1d7520fe81c66bb7377f528bdb8ddcd Oct 30 20:42:06.052491 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:06.052458 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28cdc84e-kl2cb"] Oct 30 20:42:06.055779 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:06.055755 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28cdc84e-kl2cb"] Oct 30 20:42:06.506166 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:06.506136 4292 generic.go:356] "Generic (PLEG): container finished" podID="47850ba0-2841-4253-a00e-12787f69297f" containerID="f205b5116684f02591ffbabf95ce3ef5975da0fe38049c540f0b3a53065506fc" exitCode=0 Oct 30 20:42:06.506340 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:06.506223 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b0a183-k26fm" event={"ID":"47850ba0-2841-4253-a00e-12787f69297f","Type":"ContainerDied","Data":"f205b5116684f02591ffbabf95ce3ef5975da0fe38049c540f0b3a53065506fc"} Oct 30 20:42:06.506340 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:06.506280 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b0a183-k26fm" event={"ID":"47850ba0-2841-4253-a00e-12787f69297f","Type":"ContainerStarted","Data":"e85b425d14c6f4e6bd73b5a04dc538e1a1d7520fe81c66bb7377f528bdb8ddcd"} Oct 30 20:42:06.580273 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:06.580205 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fd35199-fa8d-403d-a97e-73de633b5052" path="/var/lib/kubelet/pods/3fd35199-fa8d-403d-a97e-73de633b5052/volumes" Oct 30 20:42:06.580664 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:06.580649 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbb2a8ea-5586-4dcd-a257-385a24859651" path="/var/lib/kubelet/pods/bbb2a8ea-5586-4dcd-a257-385a24859651/volumes" Oct 30 20:42:07.635106 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:07.635083 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b0a183-k26fm" Oct 30 20:42:07.751664 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:07.751633 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47850ba0-2841-4253-a00e-12787f69297f-kube-api-access-nq72v\" (UniqueName: \"kubernetes.io/projected/47850ba0-2841-4253-a00e-12787f69297f-kube-api-access-nq72v\") pod \"47850ba0-2841-4253-a00e-12787f69297f\" (UID: \"47850ba0-2841-4253-a00e-12787f69297f\") " Oct 30 20:42:07.751840 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:07.751687 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47850ba0-2841-4253-a00e-12787f69297f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47850ba0-2841-4253-a00e-12787f69297f-maas-api-openshift-ca\") pod \"47850ba0-2841-4253-a00e-12787f69297f\" (UID: \"47850ba0-2841-4253-a00e-12787f69297f\") " Oct 30 20:42:07.751840 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:07.751719 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47850ba0-2841-4253-a00e-12787f69297f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47850ba0-2841-4253-a00e-12787f69297f-maas-api-xks-ca\") pod \"47850ba0-2841-4253-a00e-12787f69297f\" (UID: \"47850ba0-2841-4253-a00e-12787f69297f\") " Oct 30 20:42:07.752297 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:07.752254 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47850ba0-2841-4253-a00e-12787f69297f-maas-api-openshift-ca" pod "47850ba0-2841-4253-a00e-12787f69297f" (UID: "47850ba0-2841-4253-a00e-12787f69297f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:42:07.754535 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:07.754493 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47850ba0-2841-4253-a00e-12787f69297f-maas-api-xks-ca" pod "47850ba0-2841-4253-a00e-12787f69297f" (UID: "47850ba0-2841-4253-a00e-12787f69297f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:42:07.754742 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:07.754686 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47850ba0-2841-4253-a00e-12787f69297f-kube-api-access-nq72v" pod "47850ba0-2841-4253-a00e-12787f69297f" (UID: "47850ba0-2841-4253-a00e-12787f69297f"). InnerVolumeSpecName "kube-api-access-nq72v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:42:07.852877 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:07.852846 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nq72v\" (UniqueName: \"kubernetes.io/projected/47850ba0-2841-4253-a00e-12787f69297f-kube-api-access-nq72v\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:42:07.852877 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:07.852874 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47850ba0-2841-4253-a00e-12787f69297f-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:42:07.852877 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:07.852886 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47850ba0-2841-4253-a00e-12787f69297f-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:42:08.516292 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:08.516266 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b0a183-k26fm" Oct 30 20:42:08.516484 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:08.516310 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b0a183-k26fm" event={"ID":"47850ba0-2841-4253-a00e-12787f69297f","Type":"ContainerDied","Data":"e85b425d14c6f4e6bd73b5a04dc538e1a1d7520fe81c66bb7377f528bdb8ddcd"} Oct 30 20:42:08.516484 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:08.516378 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e85b425d14c6f4e6bd73b5a04dc538e1a1d7520fe81c66bb7377f528bdb8ddcd" Oct 30 20:42:08.992010 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:08.991984 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk" Oct 30 20:42:09.166182 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:09.166108 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f-credential-socket\") pod \"05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f\" (UID: \"05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f\") " Oct 30 20:42:09.166182 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:09.166178 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f-istiod-ca-cert\") pod \"05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f\" (UID: \"05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f\") " Oct 30 20:42:09.166423 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:09.166218 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f-istio-token\" (UniqueName: \"kubernetes.io/projected/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f-istio-token\") pod \"05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f\" (UID: \"05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f\") " Oct 30 20:42:09.166423 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:09.166274 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f-istio-data\") pod \"05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f\" (UID: \"05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f\") " Oct 30 20:42:09.166423 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:09.166325 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f-workload-socket\") pod \"05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f\" (UID: \"05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f\") " Oct 30 20:42:09.166423 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:09.166377 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f-kube-api-access-xrr2q\" (UniqueName: \"kubernetes.io/projected/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f-kube-api-access-xrr2q\") pod \"05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f\" (UID: \"05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f\") " Oct 30 20:42:09.166665 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:09.166441 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f-credential-socket" pod "05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f" (UID: "05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:42:09.166665 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:09.166454 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f-istio-podinfo\") pod \"05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f\" (UID: \"05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f\") " Oct 30 20:42:09.166665 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:09.166539 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f-istio-envoy\") pod \"05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f\" (UID: \"05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f\") " Oct 30 20:42:09.166665 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:09.166601 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f-workload-certs\") pod \"05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f\" (UID: \"05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f\") " Oct 30 20:42:09.166665 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:09.166627 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f-istio-data" pod "05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f" (UID: "05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:42:09.166973 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:09.166664 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f-workload-socket" pod "05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f" (UID: "05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:42:09.166973 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:09.166675 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f-istiod-ca-cert" pod "05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f" (UID: "05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:42:09.166973 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:09.166866 4292 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f-istiod-ca-cert\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:42:09.166973 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:09.166884 4292 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f-istio-data\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:42:09.166973 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:09.166892 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f-workload-certs" pod "05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f" (UID: "05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:42:09.166973 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:09.166897 4292 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f-workload-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:42:09.166973 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:09.166928 4292 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f-credential-socket\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:42:09.168529 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:09.168509 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f-istio-envoy" pod "05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f" (UID: "05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:42:09.168664 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:09.168644 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f-istio-token" pod "05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f" (UID: "05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:42:09.168929 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:09.168914 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f-kube-api-access-xrr2q" pod "05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f" (UID: "05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f"). InnerVolumeSpecName "kube-api-access-xrr2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:42:09.168974 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:09.168924 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f-istio-podinfo" pod "05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f" (UID: "05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:42:09.267373 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:09.267317 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrr2q\" (UniqueName: \"kubernetes.io/projected/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f-kube-api-access-xrr2q\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:42:09.267373 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:09.267374 4292 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f-istio-podinfo\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:42:09.267580 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:09.267386 4292 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f-istio-envoy\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:42:09.267580 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:09.267395 4292 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f-workload-certs\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:42:09.267580 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:09.267403 4292 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f-istio-token\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:42:09.522577 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:09.522530 4292 generic.go:356] "Generic (PLEG): container finished" podID="05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f" containerID="fefa5b1bed1324ee25a7037a3d40050b2cc56acdb453ca4017422905c1a59191" exitCode=0 Oct 30 20:42:09.522776 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:09.522600 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk" Oct 30 20:42:09.522776 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:09.522621 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk" event={"ID":"05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f","Type":"ContainerDied","Data":"fefa5b1bed1324ee25a7037a3d40050b2cc56acdb453ca4017422905c1a59191"} Oct 30 20:42:09.522776 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:09.522681 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk" event={"ID":"05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f","Type":"ContainerDied","Data":"ec9142b42e78b41fbe1a4add63e1ff16bd376771c5477e65b0effe258264ba9c"} Oct 30 20:42:09.522776 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:09.522705 4292 scope.go:122] "RemoveContainer" containerID="fefa5b1bed1324ee25a7037a3d40050b2cc56acdb453ca4017422905c1a59191" Oct 30 20:42:09.531745 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:09.531729 4292 scope.go:122] "RemoveContainer" containerID="d2f11e436c2c3f19ddda6c0d550ba52cb78b132fad4220f471509dfe1bbb333f" Oct 30 20:42:09.539654 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:09.539637 4292 scope.go:122] "RemoveContainer" containerID="fefa5b1bed1324ee25a7037a3d40050b2cc56acdb453ca4017422905c1a59191" Oct 30 20:42:09.539930 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:42:09.539909 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fefa5b1bed1324ee25a7037a3d40050b2cc56acdb453ca4017422905c1a59191\": container with ID starting with fefa5b1bed1324ee25a7037a3d40050b2cc56acdb453ca4017422905c1a59191 not found: ID does not exist" containerID="fefa5b1bed1324ee25a7037a3d40050b2cc56acdb453ca4017422905c1a59191" Oct 30 20:42:09.539978 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:09.539939 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fefa5b1bed1324ee25a7037a3d40050b2cc56acdb453ca4017422905c1a59191"} err="failed to get container status \"fefa5b1bed1324ee25a7037a3d40050b2cc56acdb453ca4017422905c1a59191\": rpc error: code = NotFound desc = could not find container \"fefa5b1bed1324ee25a7037a3d40050b2cc56acdb453ca4017422905c1a59191\": container with ID starting with fefa5b1bed1324ee25a7037a3d40050b2cc56acdb453ca4017422905c1a59191 not found: ID does not exist" Oct 30 20:42:09.539978 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:09.539955 4292 scope.go:122] "RemoveContainer" containerID="d2f11e436c2c3f19ddda6c0d550ba52cb78b132fad4220f471509dfe1bbb333f" Oct 30 20:42:09.540177 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:42:09.540159 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2f11e436c2c3f19ddda6c0d550ba52cb78b132fad4220f471509dfe1bbb333f\": container with ID starting with d2f11e436c2c3f19ddda6c0d550ba52cb78b132fad4220f471509dfe1bbb333f not found: ID does not exist" containerID="d2f11e436c2c3f19ddda6c0d550ba52cb78b132fad4220f471509dfe1bbb333f" Oct 30 20:42:09.540236 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:09.540185 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2f11e436c2c3f19ddda6c0d550ba52cb78b132fad4220f471509dfe1bbb333f"} err="failed to get container status \"d2f11e436c2c3f19ddda6c0d550ba52cb78b132fad4220f471509dfe1bbb333f\": rpc error: code = NotFound desc = could not find container \"d2f11e436c2c3f19ddda6c0d550ba52cb78b132fad4220f471509dfe1bbb333f\": container with ID starting with d2f11e436c2c3f19ddda6c0d550ba52cb78b132fad4220f471509dfe1bbb333f not found: ID does not exist" Oct 30 20:42:09.691255 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:09.691224 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk"] Oct 30 20:42:09.742062 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:09.742019 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-64120b-openshift-default-6dff766d79-slxqk"] Oct 30 20:42:10.580318 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:10.580224 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f" path="/var/lib/kubelet/pods/05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f/volumes" Oct 30 20:42:13.219815 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:42:13.219775 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:42:16.370729 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:16.370690 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-b0a183-5f4f79ff84-pwfs7"] Oct 30 20:42:21.371163 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:21.371103 4292 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-b0a183-5f4f79ff84-pwfs7" podUID="7c69db71-a50d-4477-bd35-1eb6c5e47e62" containerName="payload-processing" containerID="cri-o://5e03571e037122c20cfa35502055b8efec286ae74f4dc4ca7711964ed74df684" gracePeriod=25 Oct 30 20:42:22.044727 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:22.044694 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e46c610b-hc6w9"] Oct 30 20:42:22.044999 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:22.044988 4292 cpu_manager.go:414] "containerMap: removing container" podUID="05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f" containerName="istio-proxy" Oct 30 20:42:22.045052 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:22.045000 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f" containerName="istio-proxy" Oct 30 20:42:22.045052 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:22.045011 4292 cpu_manager.go:414] "containerMap: removing container" podUID="47850ba0-2841-4253-a00e-12787f69297f" containerName="revoke-keys" Oct 30 20:42:22.045052 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:22.045015 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47850ba0-2841-4253-a00e-12787f69297f" containerName="revoke-keys" Oct 30 20:42:22.045052 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:22.045029 4292 cpu_manager.go:414] "containerMap: removing container" podUID="05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f" containerName="istio-proxy" Oct 30 20:42:22.045052 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:22.045034 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f" containerName="istio-proxy" Oct 30 20:42:22.045202 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:22.045082 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f" containerName="istio-proxy" Oct 30 20:42:22.045202 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:22.045093 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="47850ba0-2841-4253-a00e-12787f69297f" containerName="revoke-keys" Oct 30 20:42:22.048129 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:22.048115 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e46c610b-hc6w9" Oct 30 20:42:22.070639 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:22.070608 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4d82e82-28a9-4c06-b69a-b46189af7c07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e46c610b-hc6w9\" (UID: \"c4d82e82-28a9-4c06-b69a-b46189af7c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e46c610b-hc6w9" Oct 30 20:42:22.070639 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:22.070654 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crmn6\" (UniqueName: \"kubernetes.io/projected/c4d82e82-28a9-4c06-b69a-b46189af7c07-kube-api-access-crmn6\") pod \"maas-api-revoke-sub-e46c610b-hc6w9\" (UID: \"c4d82e82-28a9-4c06-b69a-b46189af7c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e46c610b-hc6w9" Oct 30 20:42:22.070851 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:22.070760 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4d82e82-28a9-4c06-b69a-b46189af7c07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e46c610b-hc6w9\" (UID: \"c4d82e82-28a9-4c06-b69a-b46189af7c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e46c610b-hc6w9" Oct 30 20:42:22.103735 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:22.103705 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e46c610b-hc6w9"] Oct 30 20:42:22.152978 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:22.152951 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:42:22.153126 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:22.152992 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:42:22.153126 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:22.152992 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:42:22.154687 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:22.154674 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\"" Oct 30 20:42:22.171640 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:22.171617 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4d82e82-28a9-4c06-b69a-b46189af7c07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e46c610b-hc6w9\" (UID: \"c4d82e82-28a9-4c06-b69a-b46189af7c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e46c610b-hc6w9" Oct 30 20:42:22.171716 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:22.171677 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4d82e82-28a9-4c06-b69a-b46189af7c07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e46c610b-hc6w9\" (UID: \"c4d82e82-28a9-4c06-b69a-b46189af7c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e46c610b-hc6w9" Oct 30 20:42:22.171763 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:22.171719 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crmn6\" (UniqueName: \"kubernetes.io/projected/c4d82e82-28a9-4c06-b69a-b46189af7c07-kube-api-access-crmn6\") pod \"maas-api-revoke-sub-e46c610b-hc6w9\" (UID: \"c4d82e82-28a9-4c06-b69a-b46189af7c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e46c610b-hc6w9" Oct 30 20:42:22.172343 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:22.172327 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4d82e82-28a9-4c06-b69a-b46189af7c07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e46c610b-hc6w9\" (UID: \"c4d82e82-28a9-4c06-b69a-b46189af7c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e46c610b-hc6w9" Oct 30 20:42:22.174247 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:22.174228 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4d82e82-28a9-4c06-b69a-b46189af7c07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e46c610b-hc6w9\" (UID: \"c4d82e82-28a9-4c06-b69a-b46189af7c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e46c610b-hc6w9" Oct 30 20:42:22.244220 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:22.244184 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crmn6\" (UniqueName: \"kubernetes.io/projected/c4d82e82-28a9-4c06-b69a-b46189af7c07-kube-api-access-crmn6\") pod \"maas-api-revoke-sub-e46c610b-hc6w9\" (UID: \"c4d82e82-28a9-4c06-b69a-b46189af7c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e46c610b-hc6w9" Oct 30 20:42:22.360668 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:22.360598 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e46c610b-hc6w9" Oct 30 20:42:22.740126 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:22.740102 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e46c610b-hc6w9"] Oct 30 20:42:22.742158 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:42:22.742134 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4d82e82_28a9_4c06_b69a_b46189af7c07.slice/crio-e7a79de6a2dc63686947fa92e61f7bc0cf8bef0e578579c967d6170307f1e904 WatchSource:0}: Error finding container e7a79de6a2dc63686947fa92e61f7bc0cf8bef0e578579c967d6170307f1e904: Status 404 returned error can't find the container with id e7a79de6a2dc63686947fa92e61f7bc0cf8bef0e578579c967d6170307f1e904 Oct 30 20:42:23.591755 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:23.591727 4292 generic.go:356] "Generic (PLEG): container finished" podID="c4d82e82-28a9-4c06-b69a-b46189af7c07" containerID="41c85dba2b46f484df25e3c74167d8ef13824281c378ad0e519b8dae9c0f9a3e" exitCode=0 Oct 30 20:42:23.591880 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:23.591784 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e46c610b-hc6w9" event={"ID":"c4d82e82-28a9-4c06-b69a-b46189af7c07","Type":"ContainerDied","Data":"41c85dba2b46f484df25e3c74167d8ef13824281c378ad0e519b8dae9c0f9a3e"} Oct 30 20:42:23.591880 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:23.591828 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e46c610b-hc6w9" event={"ID":"c4d82e82-28a9-4c06-b69a-b46189af7c07","Type":"ContainerStarted","Data":"e7a79de6a2dc63686947fa92e61f7bc0cf8bef0e578579c967d6170307f1e904"} Oct 30 20:42:24.721745 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:24.721718 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e46c610b-hc6w9" Oct 30 20:42:24.794687 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:24.794658 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4d82e82-28a9-4c06-b69a-b46189af7c07-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4d82e82-28a9-4c06-b69a-b46189af7c07-maas-api-openshift-ca\") pod \"c4d82e82-28a9-4c06-b69a-b46189af7c07\" (UID: \"c4d82e82-28a9-4c06-b69a-b46189af7c07\") " Oct 30 20:42:24.794884 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:24.794721 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4d82e82-28a9-4c06-b69a-b46189af7c07-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4d82e82-28a9-4c06-b69a-b46189af7c07-maas-api-xks-ca\") pod \"c4d82e82-28a9-4c06-b69a-b46189af7c07\" (UID: \"c4d82e82-28a9-4c06-b69a-b46189af7c07\") " Oct 30 20:42:24.794884 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:24.794799 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4d82e82-28a9-4c06-b69a-b46189af7c07-kube-api-access-crmn6\" (UniqueName: \"kubernetes.io/projected/c4d82e82-28a9-4c06-b69a-b46189af7c07-kube-api-access-crmn6\") pod \"c4d82e82-28a9-4c06-b69a-b46189af7c07\" (UID: \"c4d82e82-28a9-4c06-b69a-b46189af7c07\") " Oct 30 20:42:24.795060 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:24.795028 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4d82e82-28a9-4c06-b69a-b46189af7c07-maas-api-openshift-ca" pod "c4d82e82-28a9-4c06-b69a-b46189af7c07" (UID: "c4d82e82-28a9-4c06-b69a-b46189af7c07"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:42:24.796818 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:24.796791 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4d82e82-28a9-4c06-b69a-b46189af7c07-maas-api-xks-ca" pod "c4d82e82-28a9-4c06-b69a-b46189af7c07" (UID: "c4d82e82-28a9-4c06-b69a-b46189af7c07"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:42:24.797160 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:24.797144 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4d82e82-28a9-4c06-b69a-b46189af7c07-kube-api-access-crmn6" pod "c4d82e82-28a9-4c06-b69a-b46189af7c07" (UID: "c4d82e82-28a9-4c06-b69a-b46189af7c07"). InnerVolumeSpecName "kube-api-access-crmn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:42:24.896261 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:24.896188 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4d82e82-28a9-4c06-b69a-b46189af7c07-maas-api-xks-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:42:24.896261 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:24.896216 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crmn6\" (UniqueName: \"kubernetes.io/projected/c4d82e82-28a9-4c06-b69a-b46189af7c07-kube-api-access-crmn6\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:42:24.896261 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:24.896227 4292 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4d82e82-28a9-4c06-b69a-b46189af7c07-maas-api-openshift-ca\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:42:25.025390 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:25.025305 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-b0a183-5f4f79ff84-pwfs7" podUID="7c69db71-a50d-4477-bd35-1eb6c5e47e62" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:42:25.601028 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:25.601001 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e46c610b-hc6w9" Oct 30 20:42:25.601028 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:25.601021 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e46c610b-hc6w9" event={"ID":"c4d82e82-28a9-4c06-b69a-b46189af7c07","Type":"ContainerDied","Data":"e7a79de6a2dc63686947fa92e61f7bc0cf8bef0e578579c967d6170307f1e904"} Oct 30 20:42:25.601234 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:25.601060 4292 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7a79de6a2dc63686947fa92e61f7bc0cf8bef0e578579c967d6170307f1e904" Oct 30 20:42:35.025275 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:35.025236 4292 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-b0a183-5f4f79ff84-pwfs7" podUID="7c69db71-a50d-4477-bd35-1eb6c5e47e62" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:42:39.577980 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:39.577942 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b0a183-k26fm"] Oct 30 20:42:39.628551 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:39.628524 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b0a183-k26fm"] Oct 30 20:42:40.580368 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:40.580331 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47850ba0-2841-4253-a00e-12787f69297f" path="/var/lib/kubelet/pods/47850ba0-2841-4253-a00e-12787f69297f/volumes" Oct 30 20:42:41.608449 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:41.608425 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-b0a183-5f4f79ff84-pwfs7" Oct 30 20:42:41.675408 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:41.675328 4292 generic.go:356] "Generic (PLEG): container finished" podID="7c69db71-a50d-4477-bd35-1eb6c5e47e62" containerID="5e03571e037122c20cfa35502055b8efec286ae74f4dc4ca7711964ed74df684" exitCode=0 Oct 30 20:42:41.675408 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:41.675384 4292 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-b0a183-5f4f79ff84-pwfs7" Oct 30 20:42:41.675408 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:41.675385 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-b0a183-5f4f79ff84-pwfs7" event={"ID":"7c69db71-a50d-4477-bd35-1eb6c5e47e62","Type":"ContainerDied","Data":"5e03571e037122c20cfa35502055b8efec286ae74f4dc4ca7711964ed74df684"} Oct 30 20:42:41.675622 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:41.675445 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-b0a183-5f4f79ff84-pwfs7" event={"ID":"7c69db71-a50d-4477-bd35-1eb6c5e47e62","Type":"ContainerDied","Data":"34c97823015340820af43737f32dacaec5ae08a2f75e3bfe47c76cb1b09ac341"} Oct 30 20:42:41.675622 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:41.675470 4292 scope.go:122] "RemoveContainer" containerID="5e03571e037122c20cfa35502055b8efec286ae74f4dc4ca7711964ed74df684" Oct 30 20:42:41.683569 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:41.683553 4292 scope.go:122] "RemoveContainer" containerID="5e03571e037122c20cfa35502055b8efec286ae74f4dc4ca7711964ed74df684" Oct 30 20:42:41.683821 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:42:41.683800 4292 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e03571e037122c20cfa35502055b8efec286ae74f4dc4ca7711964ed74df684\": container with ID starting with 5e03571e037122c20cfa35502055b8efec286ae74f4dc4ca7711964ed74df684 not found: ID does not exist" containerID="5e03571e037122c20cfa35502055b8efec286ae74f4dc4ca7711964ed74df684" Oct 30 20:42:41.683894 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:41.683835 4292 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e03571e037122c20cfa35502055b8efec286ae74f4dc4ca7711964ed74df684"} err="failed to get container status \"5e03571e037122c20cfa35502055b8efec286ae74f4dc4ca7711964ed74df684\": rpc error: code = NotFound desc = could not find container \"5e03571e037122c20cfa35502055b8efec286ae74f4dc4ca7711964ed74df684\": container with ID starting with 5e03571e037122c20cfa35502055b8efec286ae74f4dc4ca7711964ed74df684 not found: ID does not exist" Oct 30 20:42:41.732077 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:41.732047 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c69db71-a50d-4477-bd35-1eb6c5e47e62-kube-api-access-72vbh\" (UniqueName: \"kubernetes.io/projected/7c69db71-a50d-4477-bd35-1eb6c5e47e62-kube-api-access-72vbh\") pod \"7c69db71-a50d-4477-bd35-1eb6c5e47e62\" (UID: \"7c69db71-a50d-4477-bd35-1eb6c5e47e62\") " Oct 30 20:42:41.732214 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:41.732175 4292 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c69db71-a50d-4477-bd35-1eb6c5e47e62-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c69db71-a50d-4477-bd35-1eb6c5e47e62-plugins-config-volume\") pod \"7c69db71-a50d-4477-bd35-1eb6c5e47e62\" (UID: \"7c69db71-a50d-4477-bd35-1eb6c5e47e62\") " Oct 30 20:42:41.732553 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:41.732528 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c69db71-a50d-4477-bd35-1eb6c5e47e62-plugins-config-volume" pod "7c69db71-a50d-4477-bd35-1eb6c5e47e62" (UID: "7c69db71-a50d-4477-bd35-1eb6c5e47e62"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:42:41.734081 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:41.734059 4292 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c69db71-a50d-4477-bd35-1eb6c5e47e62-kube-api-access-72vbh" pod "7c69db71-a50d-4477-bd35-1eb6c5e47e62" (UID: "7c69db71-a50d-4477-bd35-1eb6c5e47e62"). InnerVolumeSpecName "kube-api-access-72vbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:42:41.833294 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:41.833260 4292 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72vbh\" (UniqueName: \"kubernetes.io/projected/7c69db71-a50d-4477-bd35-1eb6c5e47e62-kube-api-access-72vbh\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:42:41.833294 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:41.833286 4292 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c69db71-a50d-4477-bd35-1eb6c5e47e62-plugins-config-volume\") on node \"ip-10-0-131-147.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:42:42.099513 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:42.099479 4292 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-b0a183-5f4f79ff84-pwfs7"] Oct 30 20:42:42.106125 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:42.106100 4292 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-b0a183-5f4f79ff84-pwfs7"] Oct 30 20:42:42.581408 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:42.581341 4292 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c69db71-a50d-4477-bd35-1eb6c5e47e62" path="/var/lib/kubelet/pods/7c69db71-a50d-4477-bd35-1eb6c5e47e62/volumes" Oct 30 20:42:43.218655 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:42:43.218617 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:42:50.757832 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:50.757803 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-q9kwv_90740ed8-ad1f-42ac-8e4c-4db19bde978a/kuadrant-console-plugin/0.log" Oct 30 20:42:53.661120 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:53.661093 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-dshrx_896347de-fc5a-480a-a436-0d2f26b0b76c/manager/0.log" Oct 30 20:42:53.914617 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:53.914529 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-5hwsq_371c7b75-70f3-4eb8-8ae0-dca9232838cb/manager/0.log" Oct 30 20:42:54.336757 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:54.336727 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7d7f7cf6c4-bcgck_488cec0c-d7a1-4adb-a8c9-923c0072b88d/manager/0.log" Oct 30 20:42:55.051128 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:55.051095 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-ljppk_8f379085-d64c-422c-88e3-b81271212343/proxy/0.log" Oct 30 20:42:58.584002 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:58.583967 4292 scope.go:122] "RemoveContainer" containerID="72203eb220235d5802b6bb22ec9ded1ff88c64dc00c9d3f8534399e365faf983" Oct 30 20:42:58.592339 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:58.592318 4292 scope.go:122] "RemoveContainer" containerID="244537e2840f179f396186fe7db08ed4fdacf335c1701b0cd0b2a534cff4af95" Oct 30 20:42:58.599750 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:58.599733 4292 scope.go:122] "RemoveContainer" containerID="0002a3d2eead2a497cdf9eec96246ae98340c7e38e7fb2b6c0d14b594cec8dce" Oct 30 20:42:58.607152 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:58.607136 4292 scope.go:122] "RemoveContainer" containerID="17d219b2d958ad656c253317956378eb40dea1e37e9aaa356780455f4586c596" Oct 30 20:42:58.614153 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:42:58.614138 4292 scope.go:122] "RemoveContainer" containerID="fde6c50b57d144f647ed77f71672111dc42f5f62b8b27e5a50710a4b547af5aa" Oct 30 20:43:06.723897 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:06.723859 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-dshrx_896347de-fc5a-480a-a436-0d2f26b0b76c/manager/0.log" Oct 30 20:43:06.981598 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:06.981516 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-5hwsq_371c7b75-70f3-4eb8-8ae0-dca9232838cb/manager/0.log" Oct 30 20:43:07.417590 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:07.417553 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7d7f7cf6c4-bcgck_488cec0c-d7a1-4adb-a8c9-923c0072b88d/manager/0.log" Oct 30 20:43:08.122484 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:08.122456 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-ljppk_8f379085-d64c-422c-88e3-b81271212343/proxy/0.log" Oct 30 20:43:09.184004 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:09.183975 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-1301e352-zpl5b_9ac38922-9d8d-4005-b611-1f129ccc4f69/revoke-keys/0.log" Oct 30 20:43:09.412638 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:09.412607 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b75042a-jbssd_30ae8747-6c31-4160-8548-1a80e5305294/revoke-keys/0.log" Oct 30 20:43:09.868045 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:09.868011 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-997c20ab-h8thk_160c9418-9850-4c39-8691-55202031b0a4/revoke-keys/0.log" Oct 30 20:43:10.090170 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:10.090141 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-e46c610b-hc6w9_c4d82e82-28a9-4c06-b69a-b46189af7c07/revoke-keys/0.log" Oct 30 20:43:12.522208 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:12.522179 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-q9kwv_90740ed8-ad1f-42ac-8e4c-4db19bde978a/kuadrant-console-plugin/0.log" Oct 30 20:43:13.218580 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:43:13.218534 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:43:14.890979 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:14.890947 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-b9md6_97288e93-eff2-4d48-8cc5-78588579066a/discovery/0.log" Oct 30 20:43:15.128074 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:15.128047 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5fc9b9cc5d-nft4n_9e8dccba-2e77-4679-bed8-4ee069155583/kube-auth-proxy/0.log" Oct 30 20:43:29.192056 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:29.192016 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-flnvg/must-gather-js92z"] Oct 30 20:43:29.192602 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:29.192464 4292 cpu_manager.go:414] "containerMap: removing container" podUID="7c69db71-a50d-4477-bd35-1eb6c5e47e62" containerName="payload-processing" Oct 30 20:43:29.192602 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:29.192482 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c69db71-a50d-4477-bd35-1eb6c5e47e62" containerName="payload-processing" Oct 30 20:43:29.192602 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:29.192502 4292 cpu_manager.go:414] "containerMap: removing container" podUID="c4d82e82-28a9-4c06-b69a-b46189af7c07" containerName="revoke-keys" Oct 30 20:43:29.192602 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:29.192510 4292 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4d82e82-28a9-4c06-b69a-b46189af7c07" containerName="revoke-keys" Oct 30 20:43:29.192602 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:29.192576 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="05fc41dc-e6b5-4d5a-8a1d-cd1c3242f07f" containerName="istio-proxy" Oct 30 20:43:29.192602 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:29.192588 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4d82e82-28a9-4c06-b69a-b46189af7c07" containerName="revoke-keys" Oct 30 20:43:29.192602 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:29.192602 4292 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c69db71-a50d-4477-bd35-1eb6c5e47e62" containerName="payload-processing" Oct 30 20:43:29.195719 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:29.195698 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-flnvg/must-gather-js92z" Oct 30 20:43:29.248446 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:29.248416 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-flnvg/must-gather-js92z"] Oct 30 20:43:29.297641 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:29.297612 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-flnvg\"/\"default-dockercfg-s99rd\"" Oct 30 20:43:29.300470 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:29.300452 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-flnvg\"/\"openshift-service-ca.crt\"" Oct 30 20:43:29.300534 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:29.300459 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-flnvg\"/\"kube-root-ca.crt\"" Oct 30 20:43:29.323837 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:29.323810 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e7c91260-564d-438c-9fd5-6f7279fb5eb8-must-gather-output\") pod \"must-gather-js92z\" (UID: \"e7c91260-564d-438c-9fd5-6f7279fb5eb8\") " pod="openshift-must-gather-flnvg/must-gather-js92z" Oct 30 20:43:29.323976 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:29.323846 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxpsg\" (UniqueName: \"kubernetes.io/projected/e7c91260-564d-438c-9fd5-6f7279fb5eb8-kube-api-access-qxpsg\") pod \"must-gather-js92z\" (UID: \"e7c91260-564d-438c-9fd5-6f7279fb5eb8\") " pod="openshift-must-gather-flnvg/must-gather-js92z" Oct 30 20:43:29.425154 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:29.425127 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e7c91260-564d-438c-9fd5-6f7279fb5eb8-must-gather-output\") pod \"must-gather-js92z\" (UID: \"e7c91260-564d-438c-9fd5-6f7279fb5eb8\") " pod="openshift-must-gather-flnvg/must-gather-js92z" Oct 30 20:43:29.425327 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:29.425170 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxpsg\" (UniqueName: \"kubernetes.io/projected/e7c91260-564d-438c-9fd5-6f7279fb5eb8-kube-api-access-qxpsg\") pod \"must-gather-js92z\" (UID: \"e7c91260-564d-438c-9fd5-6f7279fb5eb8\") " pod="openshift-must-gather-flnvg/must-gather-js92z" Oct 30 20:43:29.425545 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:29.425524 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e7c91260-564d-438c-9fd5-6f7279fb5eb8-must-gather-output\") pod \"must-gather-js92z\" (UID: \"e7c91260-564d-438c-9fd5-6f7279fb5eb8\") " pod="openshift-must-gather-flnvg/must-gather-js92z" Oct 30 20:43:29.490174 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:29.490110 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxpsg\" (UniqueName: \"kubernetes.io/projected/e7c91260-564d-438c-9fd5-6f7279fb5eb8-kube-api-access-qxpsg\") pod \"must-gather-js92z\" (UID: \"e7c91260-564d-438c-9fd5-6f7279fb5eb8\") " pod="openshift-must-gather-flnvg/must-gather-js92z" Oct 30 20:43:29.509034 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:29.509000 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-flnvg/must-gather-js92z" Oct 30 20:43:29.888233 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:29.888186 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-flnvg/must-gather-js92z"] Oct 30 20:43:29.897192 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:43:29.897157 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7c91260_564d_438c_9fd5_6f7279fb5eb8.slice/crio-bbe7e5b216eb9922b1f397ff52ab0d02474a9053fb0dc0b90c5bcf130ec5e1ba WatchSource:0}: Error finding container bbe7e5b216eb9922b1f397ff52ab0d02474a9053fb0dc0b90c5bcf130ec5e1ba: Status 404 returned error can't find the container with id bbe7e5b216eb9922b1f397ff52ab0d02474a9053fb0dc0b90c5bcf130ec5e1ba Oct 30 20:43:29.898935 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:29.898920 4292 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 20:43:30.897524 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:30.897487 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-flnvg/must-gather-js92z" event={"ID":"e7c91260-564d-438c-9fd5-6f7279fb5eb8","Type":"ContainerStarted","Data":"bbe7e5b216eb9922b1f397ff52ab0d02474a9053fb0dc0b90c5bcf130ec5e1ba"} Oct 30 20:43:39.099826 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:39.099799 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-kcr6x_e6aabcd8-9a8c-4e92-9f6f-5b001cea8e87/global-pull-secret-syncer/0.log" Oct 30 20:43:40.947131 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:40.947098 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-flnvg/must-gather-js92z" event={"ID":"e7c91260-564d-438c-9fd5-6f7279fb5eb8","Type":"ContainerStarted","Data":"5e68067e30a7b5d367c3b7b1795da819e4f4a4c0f3006edec4fee4653272027f"} Oct 30 20:43:40.947594 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:40.947140 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-flnvg/must-gather-js92z" event={"ID":"e7c91260-564d-438c-9fd5-6f7279fb5eb8","Type":"ContainerStarted","Data":"deed905cf492a49ccc500b2e7ae6b6f59792f929952efddfee41510b8489db36"} Oct 30 20:43:41.065687 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:41.065633 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-flnvg/must-gather-js92z" podStartSLOduration=1.890937756 podStartE2EDuration="12.065618386s" totalImagesPullingTime="10.17468063s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:43:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:43:41.064785216 +0000 UTC m=+4724.583100671" watchObservedRunningTime="2026-10-30 20:43:41.065618386 +0000 UTC m=+4724.583933843" Oct 30 20:43:41.881259 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:41.881233 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-dshrx_896347de-fc5a-480a-a436-0d2f26b0b76c/manager/0.log" Oct 30 20:43:42.330090 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:42.330058 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-5hwsq_371c7b75-70f3-4eb8-8ae0-dca9232838cb/manager/0.log" Oct 30 20:43:42.605259 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:42.605180 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7d7f7cf6c4-bcgck_488cec0c-d7a1-4adb-a8c9-923c0072b88d/manager/0.log" Oct 30 20:43:42.716989 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:42.716963 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-l66t6_c9eecef1-2e58-4444-8a51-25b0b18715f5/konnectivity-agent/0.log" Oct 30 20:43:43.221587 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:43:43.221554 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:43:43.969765 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:43.969738 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-147.us-west-2.compute.internal_799ed54bc75d16c836f178a66c89fbdc/haproxy/0.log" Oct 30 20:43:43.988493 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:43.988466 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-ljppk_8f379085-d64c-422c-88e3-b81271212343/proxy/0.log" Oct 30 20:43:44.677872 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:44.677822 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-dshrx_896347de-fc5a-480a-a436-0d2f26b0b76c/manager/0.log" Oct 30 20:43:45.161176 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:45.161144 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-5hwsq_371c7b75-70f3-4eb8-8ae0-dca9232838cb/manager/0.log" Oct 30 20:43:45.441530 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:45.441454 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7d7f7cf6c4-bcgck_488cec0c-d7a1-4adb-a8c9-923c0072b88d/manager/0.log" Oct 30 20:43:46.729630 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:46.729601 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-ljppk_8f379085-d64c-422c-88e3-b81271212343/proxy/0.log" Oct 30 20:43:47.510730 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:47.510699 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-dshrx_896347de-fc5a-480a-a436-0d2f26b0b76c/manager/0.log" Oct 30 20:43:47.954159 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:47.954120 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-5hwsq_371c7b75-70f3-4eb8-8ae0-dca9232838cb/manager/0.log" Oct 30 20:43:48.232652 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:48.232574 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7d7f7cf6c4-bcgck_488cec0c-d7a1-4adb-a8c9-923c0072b88d/manager/0.log" Oct 30 20:43:49.780719 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:49.780680 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-ljppk_8f379085-d64c-422c-88e3-b81271212343/proxy/0.log" Oct 30 20:43:49.886533 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:49.886504 4292 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-j795s/perf-node-gather-daemonset-7pfk5"] Oct 30 20:43:49.892266 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:49.892248 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j795s/perf-node-gather-daemonset-7pfk5" Oct 30 20:43:49.919399 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:49.919373 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-dshrx_896347de-fc5a-480a-a436-0d2f26b0b76c/manager/0.log" Oct 30 20:43:49.944674 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:49.944648 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j795s/perf-node-gather-daemonset-7pfk5"] Oct 30 20:43:49.993703 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:49.993657 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j795s\"/\"openshift-service-ca.crt\"" Oct 30 20:43:49.993846 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:49.993683 4292 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-j795s\"/\"default-dockercfg-bsc74\"" Oct 30 20:43:49.993846 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:49.993728 4292 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j795s\"/\"kube-root-ca.crt\"" Oct 30 20:43:50.003020 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:50.002984 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6abe7999-a28b-4aff-8b71-fb3309e7706c-sys\") pod \"perf-node-gather-daemonset-7pfk5\" (UID: \"6abe7999-a28b-4aff-8b71-fb3309e7706c\") " pod="openshift-must-gather-j795s/perf-node-gather-daemonset-7pfk5" Oct 30 20:43:50.003020 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:50.003016 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6abe7999-a28b-4aff-8b71-fb3309e7706c-lib-modules\") pod \"perf-node-gather-daemonset-7pfk5\" (UID: \"6abe7999-a28b-4aff-8b71-fb3309e7706c\") " pod="openshift-must-gather-j795s/perf-node-gather-daemonset-7pfk5" Oct 30 20:43:50.003233 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:50.003084 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6abe7999-a28b-4aff-8b71-fb3309e7706c-podres\") pod \"perf-node-gather-daemonset-7pfk5\" (UID: \"6abe7999-a28b-4aff-8b71-fb3309e7706c\") " pod="openshift-must-gather-j795s/perf-node-gather-daemonset-7pfk5" Oct 30 20:43:50.003233 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:50.003121 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6abe7999-a28b-4aff-8b71-fb3309e7706c-proc\") pod \"perf-node-gather-daemonset-7pfk5\" (UID: \"6abe7999-a28b-4aff-8b71-fb3309e7706c\") " pod="openshift-must-gather-j795s/perf-node-gather-daemonset-7pfk5" Oct 30 20:43:50.003233 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:50.003167 4292 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvt2d\" (UniqueName: \"kubernetes.io/projected/6abe7999-a28b-4aff-8b71-fb3309e7706c-kube-api-access-gvt2d\") pod \"perf-node-gather-daemonset-7pfk5\" (UID: \"6abe7999-a28b-4aff-8b71-fb3309e7706c\") " pod="openshift-must-gather-j795s/perf-node-gather-daemonset-7pfk5" Oct 30 20:43:50.103643 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:50.103570 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6abe7999-a28b-4aff-8b71-fb3309e7706c-sys\") pod \"perf-node-gather-daemonset-7pfk5\" (UID: \"6abe7999-a28b-4aff-8b71-fb3309e7706c\") " pod="openshift-must-gather-j795s/perf-node-gather-daemonset-7pfk5" Oct 30 20:43:50.103643 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:50.103604 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6abe7999-a28b-4aff-8b71-fb3309e7706c-lib-modules\") pod \"perf-node-gather-daemonset-7pfk5\" (UID: \"6abe7999-a28b-4aff-8b71-fb3309e7706c\") " pod="openshift-must-gather-j795s/perf-node-gather-daemonset-7pfk5" Oct 30 20:43:50.103643 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:50.103625 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6abe7999-a28b-4aff-8b71-fb3309e7706c-podres\") pod \"perf-node-gather-daemonset-7pfk5\" (UID: \"6abe7999-a28b-4aff-8b71-fb3309e7706c\") " pod="openshift-must-gather-j795s/perf-node-gather-daemonset-7pfk5" Oct 30 20:43:50.103643 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:50.103645 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6abe7999-a28b-4aff-8b71-fb3309e7706c-proc\") pod \"perf-node-gather-daemonset-7pfk5\" (UID: \"6abe7999-a28b-4aff-8b71-fb3309e7706c\") " pod="openshift-must-gather-j795s/perf-node-gather-daemonset-7pfk5" Oct 30 20:43:50.104016 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:50.103664 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6abe7999-a28b-4aff-8b71-fb3309e7706c-sys\") pod \"perf-node-gather-daemonset-7pfk5\" (UID: \"6abe7999-a28b-4aff-8b71-fb3309e7706c\") " pod="openshift-must-gather-j795s/perf-node-gather-daemonset-7pfk5" Oct 30 20:43:50.104016 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:50.103680 4292 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvt2d\" (UniqueName: \"kubernetes.io/projected/6abe7999-a28b-4aff-8b71-fb3309e7706c-kube-api-access-gvt2d\") pod \"perf-node-gather-daemonset-7pfk5\" (UID: \"6abe7999-a28b-4aff-8b71-fb3309e7706c\") " pod="openshift-must-gather-j795s/perf-node-gather-daemonset-7pfk5" Oct 30 20:43:50.104016 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:50.103733 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6abe7999-a28b-4aff-8b71-fb3309e7706c-proc\") pod \"perf-node-gather-daemonset-7pfk5\" (UID: \"6abe7999-a28b-4aff-8b71-fb3309e7706c\") " pod="openshift-must-gather-j795s/perf-node-gather-daemonset-7pfk5" Oct 30 20:43:50.104016 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:50.103744 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6abe7999-a28b-4aff-8b71-fb3309e7706c-podres\") pod \"perf-node-gather-daemonset-7pfk5\" (UID: \"6abe7999-a28b-4aff-8b71-fb3309e7706c\") " pod="openshift-must-gather-j795s/perf-node-gather-daemonset-7pfk5" Oct 30 20:43:50.104016 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:50.103734 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6abe7999-a28b-4aff-8b71-fb3309e7706c-lib-modules\") pod \"perf-node-gather-daemonset-7pfk5\" (UID: \"6abe7999-a28b-4aff-8b71-fb3309e7706c\") " pod="openshift-must-gather-j795s/perf-node-gather-daemonset-7pfk5" Oct 30 20:43:50.166912 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:50.166879 4292 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvt2d\" (UniqueName: \"kubernetes.io/projected/6abe7999-a28b-4aff-8b71-fb3309e7706c-kube-api-access-gvt2d\") pod \"perf-node-gather-daemonset-7pfk5\" (UID: \"6abe7999-a28b-4aff-8b71-fb3309e7706c\") " pod="openshift-must-gather-j795s/perf-node-gather-daemonset-7pfk5" Oct 30 20:43:50.205649 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:50.205619 4292 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j795s/perf-node-gather-daemonset-7pfk5" Oct 30 20:43:50.208262 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:50.208230 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-q9kwv_90740ed8-ad1f-42ac-8e4c-4db19bde978a/kuadrant-console-plugin/0.log" Oct 30 20:43:50.421720 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:50.421653 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-5hwsq_371c7b75-70f3-4eb8-8ae0-dca9232838cb/manager/0.log" Oct 30 20:43:50.596114 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:50.596083 4292 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j795s/perf-node-gather-daemonset-7pfk5"] Oct 30 20:43:50.599231 ip-10-0-131-147 kubenswrapper[4292]: W1030 20:43:50.599180 4292 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6abe7999_a28b_4aff_8b71_fb3309e7706c.slice/crio-e37bc47fade6f269302e4209854e41d9a180deb2d545e2739377360049333349 WatchSource:0}: Error finding container e37bc47fade6f269302e4209854e41d9a180deb2d545e2739377360049333349: Status 404 returned error can't find the container with id e37bc47fade6f269302e4209854e41d9a180deb2d545e2739377360049333349 Oct 30 20:43:50.736214 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:50.736146 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7d7f7cf6c4-bcgck_488cec0c-d7a1-4adb-a8c9-923c0072b88d/manager/0.log" Oct 30 20:43:51.001045 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:51.000959 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j795s/perf-node-gather-daemonset-7pfk5" event={"ID":"6abe7999-a28b-4aff-8b71-fb3309e7706c","Type":"ContainerStarted","Data":"6b8783c18f5170a5dd0d01d1f6af9f470e04a7c372265df8e44a793243a4589e"} Oct 30 20:43:51.001474 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:51.001106 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-j795s/perf-node-gather-daemonset-7pfk5" Oct 30 20:43:51.001474 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:51.001126 4292 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j795s/perf-node-gather-daemonset-7pfk5" event={"ID":"6abe7999-a28b-4aff-8b71-fb3309e7706c","Type":"ContainerStarted","Data":"e37bc47fade6f269302e4209854e41d9a180deb2d545e2739377360049333349"} Oct 30 20:43:51.119017 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:51.118968 4292 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-j795s/perf-node-gather-daemonset-7pfk5" podStartSLOduration=2.118955995 podStartE2EDuration="2.118955995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:43:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:43:51.118434376 +0000 UTC m=+4734.636749833" watchObservedRunningTime="2026-10-30 20:43:51.118955995 +0000 UTC m=+4734.637271450" Oct 30 20:43:52.010974 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:52.010948 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-ljppk_8f379085-d64c-422c-88e3-b81271212343/proxy/0.log" Oct 30 20:43:53.188172 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:53.188143 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-dshrx_896347de-fc5a-480a-a436-0d2f26b0b76c/manager/0.log" Oct 30 20:43:53.643543 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:53.643508 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-5hwsq_371c7b75-70f3-4eb8-8ae0-dca9232838cb/manager/0.log" Oct 30 20:43:53.928347 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:53.928275 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7d7f7cf6c4-bcgck_488cec0c-d7a1-4adb-a8c9-923c0072b88d/manager/0.log" Oct 30 20:43:55.159263 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:55.159234 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-ljppk_8f379085-d64c-422c-88e3-b81271212343/proxy/0.log" Oct 30 20:43:55.458733 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:55.458660 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-dshrx_896347de-fc5a-480a-a436-0d2f26b0b76c/manager/0.log" Oct 30 20:43:55.932250 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:55.932216 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-5hwsq_371c7b75-70f3-4eb8-8ae0-dca9232838cb/manager/0.log" Oct 30 20:43:56.225000 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:56.224923 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7d7f7cf6c4-bcgck_488cec0c-d7a1-4adb-a8c9-923c0072b88d/manager/0.log" Oct 30 20:43:57.017618 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:57.017586 4292 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-j795s/perf-node-gather-daemonset-7pfk5" Oct 30 20:43:57.448213 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:57.448114 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-ljppk_8f379085-d64c-422c-88e3-b81271212343/proxy/0.log" Oct 30 20:43:58.651543 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:58.651510 4292 scope.go:122] "RemoveContainer" containerID="48c79ca44f9975a2107c90da9fbd46728603e223e47e0991014fe4a4c1e57d2f" Oct 30 20:43:58.660330 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:58.660312 4292 scope.go:122] "RemoveContainer" containerID="2dfe376e748f0eba0083c69cda47a92257d377153a9a6c1945cc36be4770cf5c" Oct 30 20:43:58.667826 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:58.667808 4292 scope.go:122] "RemoveContainer" containerID="808b9e688a50fef843b1db9b538b1409682a4bfb7e78454304905df6d4264b4c" Oct 30 20:43:59.581800 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:59.581764 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-b9md6_97288e93-eff2-4d48-8cc5-78588579066a/discovery/0.log" Oct 30 20:43:59.688372 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:43:59.688333 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5fc9b9cc5d-nft4n_9e8dccba-2e77-4679-bed8-4ee069155583/kube-auth-proxy/0.log" Oct 30 20:44:00.715059 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:44:00.715031 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2d8dk_972b92b5-ad64-48c7-b124-28d535b4b6ac/node-exporter/0.log" Oct 30 20:44:01.140403 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:44:01.140378 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2d8dk_972b92b5-ad64-48c7-b124-28d535b4b6ac/kube-rbac-proxy/0.log" Oct 30 20:44:01.566731 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:44:01.566703 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2d8dk_972b92b5-ad64-48c7-b124-28d535b4b6ac/init-textfile/0.log" Oct 30 20:44:01.788750 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:44:01.788715 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-b9md6_97288e93-eff2-4d48-8cc5-78588579066a/discovery/0.log" Oct 30 20:44:01.895142 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:44:01.895074 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5fc9b9cc5d-nft4n_9e8dccba-2e77-4679-bed8-4ee069155583/kube-auth-proxy/0.log" Oct 30 20:44:06.730801 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:44:06.730761 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-q9kwv_90740ed8-ad1f-42ac-8e4c-4db19bde978a/kuadrant-console-plugin/0.log" Oct 30 20:44:09.006258 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:44:09.006230 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-q9kwv_90740ed8-ad1f-42ac-8e4c-4db19bde978a/kuadrant-console-plugin/0.log" Oct 30 20:44:09.082642 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:44:09.082611 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-7pqbj_a394616c-7264-42c7-9774-0e38966b6b14/prometheus-operator/0.log" Oct 30 20:44:09.531410 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:44:09.531381 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-7pqbj_a394616c-7264-42c7-9774-0e38966b6b14/kube-rbac-proxy/0.log" Oct 30 20:44:09.995043 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:44:09.994970 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-flg7z_d7e50900-a49f-4316-9d27-f58cd07fda3e/prometheus-operator-admission-webhook/0.log" Oct 30 20:44:13.222397 ip-10-0-131-147 kubenswrapper[4292]: E1030 20:44:13.222364 4292 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kwj6c" containerName="extractor" Oct 30 20:44:14.677963 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:44:14.677929 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-b9md6_97288e93-eff2-4d48-8cc5-78588579066a/discovery/0.log" Oct 30 20:44:14.793304 ip-10-0-131-147 kubenswrapper[4292]: I1030 20:44:14.793270 4292 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5fc9b9cc5d-nft4n_9e8dccba-2e77-4679-bed8-4ee069155583/kube-auth-proxy/0.log"